[terhechte]

This kind of argument, together with the "Stochastic Parrot" argument falls (in my opinion) under the category of dangerous half knowledge. Somebody read the wikipedia article on LLMs and thinks they know inside out how they work. Whereas the engineers working on these models say that they don't really know why they work / what it is that is manifesting in the different layers during training.

Yes, LLMs predict the next word, but so do we. I, at least, when I reflect on how I form sentences, also start with something and the next most likely word comes out next. You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that.

I'm not claiming that current models have feelings or a deep understanding of the subject matter. What I'm saying is that there's (high likelihood) more going on here then just the simple statistical trick that this article and others like it tend to focus on.

[fluoridation]

>You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that.

If it was purely word prediction, another person would be able to predict with 100% reliability the next word, as long as they had the same facts. The predictability is not because humans just construct sentences with no thought behind it, it's because natural language is highly redundant. A data compressor specialized in English would be able to do the same thing. Before LLMs existed there were Markov chains, and no one would argue that they had reasoning capabilities. Yet if the algorithm was stopped at "in" there was a good chance the next word would be "the", or "a", but not "in".

There's also the fact that the very purpose of language is to transmit ideas. If the next person can fill in a missing word, it's because the idea is being properly conveyed, or is not very novel. If you guess "cat" and the person says "Jupiter" instead, there's clearly miscommunication or it's something very wild.