Hacker News new | ask | show | jobs
by sillysaurusx 2371 days ago
Would you say OpenAI's dota 2 results were interesting? They're completely unreproducible according to the checklist: The dataset wasn't released, the code wasn't released, nothing but the results were released.

Ditto for AlphaGo, AlphaZero, and most other AI systems.

If one wants to argue that we shouldn't be doing AI science this way, then that's fine. It's just a different conversation.
2 comments
rckoepke 2371 days ago
LeelaChess Zero could be considered an attempt at reproducing AlphaZero.

GPT-2 (fully weighted) was arguably reproduced as well, eventually 'forcing' OpenAI to release the model they had held back [0].

0: https://www.reddit.com/r/MachineLearning/comments/by4n8p/p_s...

link
atoav 2371 days ago
So you say it could have been just a PR stunt and the AI was no AI at all?
link
sillysaurusx 2371 days ago
It's the other way around. The AI was clearly real and important, but absolutely unreproducible according to the metrics in this submission. GPT-2 is much the same: the dataset was never released, nor was the training code released. Yet would anyone say that it wasn't an important contribution to science?

I admit that it's possible the entire world is currently "doing AI wrong," though. T5 and other new projects have been better with respect to replication. But many datasets are still locked behind paywalls, or only accessible if you have an .edu email address.

link
YeGoblynQueenne 2371 days ago
Yes, I think many people would say that GPT-2 is not an important contribution to science, and that it rather is the latest focus of industry and lay press hype for machine learning and neural networks. The latest trend, if you will.

Personally, I struggle to see what new thing we learned from GPT-2. Did we learn something about the physical world? About how human minds work? About how language works? It's a language model after all. All we learned is that throwing a large dataset to a hard problem can produce results that are difficult to evaluate.

"Science" means "knowledge". If we haven't learned anything new from GPT-2 then it hasn't contributed to science. It's impressive, like a jetliner is impressive, or an aircraft carrier is impressive, but it's not increasing our body of knowledge about the world and ourselves.

link