|
This keeps emerging again an again, and the answers are pretty generic. 1. Large language models as a concept are not going anywhere anytime soon for any reason. Simply because there's no other source with a huge slice of human psyche encoded into it than the language itself and the corpus of texts in it. Humanity collectively did a massive amount of gradient descent on the language over generations, and it will stay as the primary source. That doesn't mean that other sources don't exist, of course. 2. Dataset quality matters at least as much as the architecture. There's plenty of low-hanging fruit available in preprocessing the data and "textbooks for models". You learn to count in a decimal system from both memorizing the number sequence and the explanation of the algorithm, not just by looking at millions of examples! There's plenty of a bit higher-hanging fruit available in hardware improvements and optimizations. 3. Calling a transformer a token predictor, stochastic parrot, autocomplete on steroids, etc. is of course right but kind of misses the point, like calling human brain a nerve impulse predictor (and the brain also has no "inherent way of verifying whether their predictions are correct", using the definition from the article). Reasoning about this in ill-defined terms like "understanding" or "knowledge" or "intelligence" is not useful at all. There are many differences between humans and LLMs, but the most high-level one is that humans are autonomous agents that exist in continuous time, and transformer's lifetime is the time required to compute a single token. Repeat the process for multiple tokens and you have something more complex. Add an external loopback, and you have a chatbot with memory, partly capable of doing things unexpected of a "word predictor". Make the loopback more complex, and you suddenly have an... autonomous system that exists in continuous time. Sure, it's extremely crude and primitive, and that loopback probably also needs to be replaced by something way more advanced in the future, and, and, and, and... 4. Reasoning and symbolic computation comparable to human abilities (which are also pretty spotty and error-prone) might or might not emerge as a result of scale and simple loopback mechanisms in models. You might or might not need an external symbolic engine as the author says, or maybe you can reduce it to another model of a different type, or maybe it's all wrong. Current models are still orders of magnitude smaller and simpler than the human nervous system, and plenty of things in LLMs already changed by simply increasing the scale. 5. Other than all of the above, sure - transformers or another flavor-of-the-year architecture might give way to more advanced ones. But the basic principles will remain, and language models are not going anywhere. |
This type of wording is problematic because it conflates what is written as representative of our psyche when it does not.
Psyche implies thinking and thoughts that have occurred when the brain accessed those concepts from outside our 3D world, processed it internally, vocalized it into sounds and then finally letters.
LLMs are just doing pattern text search on top of what is written, it is doing no sort of reasoning or accessing of the hyper dimensional plane like our brain does when it thinks or reasons with concepts.
Our brains are not some exotic token or neurosymbolic search engines!