[quad_eye_oh]

Right, but LLMs suggest that learning how to predict and only training on predictions is sufficient to learn anything and to have emergent generative abilities. What if learning to predict the upcoming input is all that's all that is needed for general human intelligence? What if it is all that any animals do?

[20after4]

This right here. I actually am strongly starting to believe that this is indeed what's going on.

I read the book Kingdom of Speech a few years ago and that also left me with the perspective that perhaps language has a lot more to do with how we think and perceive the world than most people like to admit. The book has been heavily criticized but I believe it made an interesting point about language.

[moffkalast]

I mean hasn't it been proven on multiple occasions that humans who never learn a language during their life also don't develop higher cognitive skills and stay on the level of a child? Clearly language plays a notable part in reasoning, and it's frankly no surprise that we're seeing this behaviour from language models.

It's also quite interesting how foundation models and fine tuning appear analogous to being born with a brain that already has six million years' worth of weights in it (trained implicitly through random changes and natural selection), which are then adapted over the course of a lifetime when environment relevant data is gradually obtained.