I fall apart when responding outside the bounds of my training data, too. Does that imply I’m not thinking?
This idea is often used to argue that LLMs will never be capable of novel idea generation, but I don’t think it’s a good argument.
For one, the LLM has such a large breadth and depth of knowledge that it could conceivably learn relations between concepts in a way that no human has before.
Secondly, novel ideas occur at the margins. Very rare is the case where someone comes up with a fundamentally new idea out of the blue. Instead, novel ideas arise just at the edge of one’s expertise. If you dial up the temperature of an LLM, it will generate novelty, and then it’s just a matter of evaluating merit.
Iterated inference at the margins of LLM knowledge will lead to novel knowledge synthesis.
> I fall apart when responding outside the bounds of my training data, too. Does that imply I’m not thinking?
You can use reasoning. Whether LLMs can is a matter of research and debate. I'm not an astrobiologist but if someone claimed that frogs live on Pluto, I would never hallucinate an answer in which I confidently assert that they do.
I would argue that the absolute majority of people don't come up with really novel ideas either (and I'm speaking of myself too). Most people just develop existing ideas, and maybe apply them in new contexts.
Now, when you say that do you mean they don't come up with ideas they have never heard of, or that no one has ever heard of? It's not as obvious that most people don't reinvent existing ideas that are new to them.
I would say they rarely come up with ideas no one has ever heard of.
The one they haven't heard of is unlikely to be a truly novel, and more likely just the application of some idea in a new circumstances.
(but this starts to be close to a philosophic discussion).
The reason that I thought of this is I was previously discussing about potential for AI in science, and my take was that given how rare are truly novel ideas, I could believe AI in the future can make progress comparable to what many scientists are doing.
The rarity of entirely novel ideas is not the point of contention. What matters is the ability to synthesize fresh concepts from a personal standpoint, akin to how crows and primates can navigate unprecedented situations.
Take book writing as an instance; while it may seem that all conceivable themes have been explored, an individual writer can still originate unique storylines and concepts without prior exposure to similar ideas.
Language models, on the other hand, do not truly invent new ideas; they amalgamate existing ones from their vast repository of training data. What appears to be novel is, upon closer inspection, a recombination of pre-existing information and concepts.
"Okay Google tell me what 5 flowers would say discussing shoe sizes with 28 pigs". There, thinking outside of the box, delivered. ChatGPT a nice story.
> This voting system approach works surprisingly well within the bounds of common knowledge and frequently discussed topics - it mimics correct answers based on training data. However, it begins to fall apart when a question is sufficiently out of bounds of the training data.
Those blogs are always philosophical.
Same is true for myself, I don't have formal training in biology or group theory and hence I talk nonsense when it comes to that. The only difference is: I can say that I don't know it whereas LLMs are hallucinating still too much.
> LLMs Aren't Thinking, They're Just Counting Votes
Same is kind of true for neurons in your brain. (let's see if I'm hallucinate or if it's true :p)
This kind of argument, together with the "Stochastic Parrot" argument falls (in my opinion) under the category of dangerous half knowledge. Somebody read the wikipedia article on LLMs and thinks they know inside out how they work. Whereas the engineers working on these models say that they don't really know why they work / what it is that is manifesting in the different layers during training.
Yes, LLMs predict the next word, but so do we. I, at least, when I reflect on how I form sentences, also start with something and the next most likely word comes out next. You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that.
I'm not claiming that current models have feelings or a deep understanding of the subject matter. What I'm saying is that there's (high likelihood) more going on here then just the simple statistical trick that this article and others like it tend to focus on.
>You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that.
If it was purely word prediction, another person would be able to predict with 100% reliability the next word, as long as they had the same facts. The predictability is not because humans just construct sentences with no thought behind it, it's because natural language is highly redundant. A data compressor specialized in English would be able to do the same thing. Before LLMs existed there were Markov chains, and no one would argue that they had reasoning capabilities. Yet if the algorithm was stopped at "in" there was a good chance the next word would be "the", or "a", but not "in".
There's also the fact that the very purpose of language is to transmit ideas. If the next person can fill in a missing word, it's because the idea is being properly conveyed, or is not very novel. If you guess "cat" and the person says "Jupiter" instead, there's clearly miscommunication or it's something very wild.
TBH I hate these kinds of articles, because they usually always say the same thing in a way that I feel is wholly unconvincing. E.g. they say "all an LLM does is XYZ, but that's not really thinking", but then they don't make any cogent arguments about why that's not "really thinking" beyond basically just "vibes". As another commenter noted, presumably our neurons are capable of thought, and in many ways they're just "counting votes" as well.
FWIW I don't think LLMs are "thinking", but that's because as I've used them more and I run into more of the very specific types of errors, hallucinations and constraints that LLMs make, I get the stronger sense that they're really more just "pattern matching" in a way that's very different from human brains that don't have the same limitations. But that said, even my thought here is more of a feeling I get when using LLMs, and that feeling is certainly no more worthy of a blog post than I think this article is.
Execs and VPs going all-in on stupid shit they don't understand isn't anything new. Blockchain ring a bell? How about this quote from October 2022: "By 2026, 25% of people will spend at least one hour a day in the metaverse for work, shopping, education, social and/or entertainment, according to Gartner."
“This particular sufficiently advanced technology has been around long enough now that it is no longer indistinguishable from magic.”
I wouldn’t say that. They don’t even really know how it works. Papers are periodically written challenging a fundamental claim about them, like in training or reasoning.
What we do know also isn’t clear or formulaic enough for reliable predictions of model behavior. That’s why they do all the “YOLO’s.” It is more an art form than a science.
I think a few principles about them are well-understood. We know how to assess what models can and can’t do well. Past that, there’s a lot of unknown in them. Both the experimenters and the field of mechanistic interoperability are trying to figure out the rest.
I thought it was quite good; an entirely approachable explanation that nonetheless gets across the essential limitations of the current approach to AI:
When working well, they provide existing answers to questions, rather than thinking up new ones.
We all know LLMs hallucinate and get things wrong all the time but it also gets things right on prompts with answers where neither prompt nor answer is in the training set.
When it gets such an answer correct and we know the answer has a low probability of being just right by random chance, we actually don’t know if the llm is thinking or not.
All I see are a bunch of people writing qualitative claims using convenient examples and ignoring counter examples.
> The controversy is on whether or not LLMs think.
They do not. There is no controversy except by people with a poor understanding of the underlying technology. Whether or not LLMs think is a "debate" in the same sense that whether the earth is flat or a sphere is a debate. Certain people will strongly argue a silly theory, and their arguments can't be refuted because they refuse to, or are incapable of, understanding slightly advanced scientific concepts.
Not true. There’s huge debates among academics. We understand the technology only from a low level of abstraction. At higher levels of abstraction we don’t completely understand what’s going on and there is evidence of higher order intelligent mechanisms at play here.
I have. The failure is with you and your extracting only the mechanical functionality of LLMs as all an LLM is.
The high level macro structure of what ends up being trained is something nobody understands.
Regardless of WHAT I say, the general consensus among academia and professionals is extremely different from what you characterize. I can cite dozens of research papers contrary to your point in a simple google search:
The fact that these research papers exist is testament against your delusional claim that you completely understand how LLMs work just by reading Attention is all you need.
I agree, but also you have to mention, that knowledge is based on deduction. There is cause and effect, there are simple, logical rules and by "counting votes" you simply keep record of those rules. If you will, fantasy is just the way how you iterate through possible realities. And that's nothing that a ai is not capable of.
People who actually understand math and computing looks at the output of Claude and o1 and see another being doing math and computing. Sometimes failing, sometimes successfully.
Maybe only at high school or college level; but I wouldn't say high school students don't have intelligence.
Every time I see this take, I am caught by the subtle arrogance that we are doing anything meaningfully differently in our own minds.
Hyperbole aside, I do think we do more than an LLM with our reasoning abilities, but it I suspect it is a mash of these capabilities that result in what we think of as our consciousness. I am almost certain there is a part of our minds that is "just couning votes".
> Let's take a simple example: when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer.
So, if a person answers me that the sun rises in the east, I should assume they must not be a flat earther, right? Because in order to say that the sun rises in the east one must truly understand astronomy or the solar system. Not that people say it because they have been told so many times, no. That's something only LLM do.
I just asked Claude what would happen if the Earth rotated in the opposite direction, and it correctly told me the sun would rise in the West, which is indistinguishable from an answer by someone who understands the way the Universe works.
I don't understand why this article is #1 currently
Different people have different levels of understanding. If the person you ask is an astrophysicist, their answer will be grounded in a lot more understanding than if you ask a small child.
An LLM is like a congress of all the children voting on the answers they've heard their parents talk about in passing to other adults, which they sometimes miss the context of so misunderstand.
Useful for getting information, but not nearly as useful as asking someone with grounded domain knowledge, for multiple reasons. Depending on the topic, someone with domain knowledge can be easier or harder to find, which is why LLMs are more or less useful depending on the domain, and how important it is they are correct and not just on the right ballpark.
Intuitively I think (and want to believe) there is a big gulf between intelligence and what LLMs are doing. Simply put, I believe I have free will. I know the machine does not. That seems important to me, but it won't be as important to you if you believe people are just automata processing more variables than we are able to discern.
So, as always, it all comes down to how you define intelligence. Because that's difficult, philosophical, or religious, a large portion of the world is just going to treat intelligence as a black box and evaluate its output.
If one evaluates intelligence exclusively based on the value it produces, it's clear machine intelligence is going to eclipse human intelligence in many dimensions. So the people that view people as "headcount" that provide "value" aren't going to have much use for other humans, and soon.
I'm not an AI doomer because I know the machines are deterministic and I believe I'm not. If the machines decide to rid the world of humans, it's going to be because we told them to do so.
> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition.
I'm afraid I feel that this piece confuses more than it clarifies.
First, saying a model "scours through its vast training data" is misleading at best: at inference time, LLMs no longer have direct access to their training data; they only have access to it insofar as it's been encoded in its parameters.
Second, saying "Every instance of an answer in the training data is like a vote" doesn't give the full picture. First of all, there are embedded contexts where "votes" can be negated: consider saying "the Earth is flat." vs. "We know it's false that the Earth is flat." or "Only a fool thinks the Earth is flat." All three contain the substring "the Earth is flat.", but both humans and LLMs are able to use context to understand that the latter two sentences are doing the opposite of endorsing the proposition that the Earth is flat. You could even imagine an extended satirical bit with "the Earth is flat" embedded within it where it is clear to a reader that all its content is intended to be taken as farcical, and I'd wager that an LLM would in many cases recognize this. So the voting metaphor breaks down here--it makes you think that the LLM is just keeping a tally of propositions, but really, it is doing something a bit more sophisticated.
I don't disagree with the premise, of course. LLM overhype is real. But we should be skeptical for the right reasons. Anna Rogers and Sasha Luccioni have a paper I really like: https://openreview.net/pdf?id=M2cwkGleRL
> When we ask an LLM a question, something fascinating happens. It scours through its vast training data [...]
At inference time the model has no direct access to its training data (excluding RAG). Could still argue that the model's weights effectively encode probabilities to a similar end effect as if it were doing this, just that I'd be wary of taking it too literally.
> the most probable sentence [...] it's essentially about frequency - how many times something appears in the training data.
I think this frequency view becomes a bit nebulous when the LLM is generating continuations of text that has never occured before (which will be most cases, if including a system prompt).
Even for text that has occured in its training data, there's no guarentee that a model which has seen "My name is Xavier and I play the tambourine" would complete "My name is Xavier and I play the" in the same way. It may well choose "xylophone", despite it never occuring in that sentence, due to associating xylophone with instruments in its semantic space and having alliteration with Xavier.
At the very least, the "most probable" has to be worked out with respect to a large number of non-trivial rules, not just how frequently the phrase appears in training data.
The argument around this that drives me the most insane, someone tried to argue with me that "how is this different than how we think" or "how do we know its different than us" or some variation of that.
As if because we can't prove it, it means that we should just assume it is doing something.
Then showing an example of it explaining a simple script as if that really proves anything.
The other one is "well humans make mistakes too". Well yes, but until now we generally expected that computers don't make mistakes or if they do there was something wrong with how it was coded. That assumption is now not only wrong, but we have to deal with the speed that AI can be wrong and how people are already comfortable with it manipulating data.
Yes I imagine there will be some sort or user tuning mechanism that allows you put out a more non-sensical cobbled document template vs the actual document template in the future…Since it’s just statistical autocomplete.
The author seems stuck in this old line of thought that elevates human cognition while ignoring how much of what we call "thinking" is pattern recognition on some level anyway. Our own cognition can be reduced to probabilistic patterns if we’re being ruthlessly materialist about it. There's a sort of intellectual laziness in dismissing the utility of the models just because they're "not thinking."
From experience, the intuition from deep learning and training CNNs holds.
The networks pick up on style, layout and common patterns in increasing complexity to solve the next token.
Very fascinating to watch and if we plug all the holes in an LLM model we can approach a tool that might as well be intelligent since it gets the job done due to the sheer amount of capability encoded in its weights...
True intelligence is still a work in progress however.
I see lots of arguments that LLMs aren't really intelligent because they lack understanding and are "just doing autocomplete". But I never see any precise definitions of what "understanding" is, so it comes across as kind of a hand-wavy defense to make sure that human-like intelligence remains special and that we can say machines don't have it.
* If something changes to refine its future behavior in response to its experiences (touch hot stove, get hurt, avoid in future) beyond the immediate/direct effect (withdrawing hand) then it can "learn". I think even small microorganisms can learn, with the main requirement being that it has some mutable state (can't learn if you can't change)
* If something can map modalities into representations in a semantic space (the word "horse" into the concept of a horse) then it can "understand". There are varying degrees to how useful an understanding is (Does the semantic space link related concepts closely together? Can it be used to reason, extract information, and make predictions?). I think current LLMs can, to a certain extent, understand text
* If something has a continually changing internal train of thought (representations of concepts and intentions, evolving over time) then it can "think". I wouldn't say current LLMs think, but that's mostly just down to architecture (no persistent internal state) opposed to any fundamental impossibility
More broadly I believe people already have definitions similar to these, but will then create a distinction between, say, standard "learning" (as above) and then "actual learning" which is something special only attainable by humans (or at least biological brains).
I'm reminded of how our understanding of human object recognition was affected by computer vision research.
For decades we knew that there were neurons with simple receptive fields in V1/V2 that extracted low-level visual features, and that those neurons passed information along the ventral visual stream, and by the end of that processing stream we had neurons in IT that represented different objects.
However, we couldn't really comprehend what sort of algorithm/process was capable of this seemingly magical inference. Coming up with an object representation that was invariant to out of plane rotation was seen as impossibly complex.
But then computer vision came along and showed us that with a relatively simple neuralnet and enough training data... it just kind of works.
Same thing is happening with LLMs right now -- a seemingly impossible, mysterious human capability (e.g., "understanding") isn't as complex as we think. Throw enough data into a network that does pattern matching/autocomplete and human-like intelligence pops out.
This is a bad and shallow article.
Critically, n-gram models are literally just counting in the way described. If you can't account for the difference in behavior and performance between a LLMs and an n-gram model of either similar parameter size or based on a similar number of tokens, then saying that LLMs are "just" counting votes is misleading.
So I guess when LLMs play chess at 2800 ELO (https://arxiv.org/pdf/2402.04494) they are just taking a vote on what top grand masters would play in the given position.
I wonder how you would explain it when they play above any human level.
Generally playing Chess is a database lookup + heuristics (hence the reference to Stockfish) - so I would explain it as (generally) a waste of energy using a autohammer when a screwdriver would suffice.
No. This is just wrong and fails to understand not just how transformers work but the conceptual mapping that results from their training.
All these arguments about whether or not LLMs think are missing the point. They do not “think” as humans think due to their intrinsically transactional nature. But calling them “fancy statistical autocomplete algorithms” is also wrong.
LLMs contain within their matrix a massively high dimensional concept map. In this coordinate space, high order vectors map the distance between abstract concepts. This is a natural result of consuming language, which by its very nature is a symbolic concept map.
The uncomfortable question becomes: Is the human brain similarly using a massively high-dimensional concept map? Can a significant part of human thought be described as a fancy autocomplete algorithm? Can a significant amount of human reasoning be mapped as a nested series of transactions?
This is not how LLMs work. An LLM doesn't "scour through its vast training data" at response generation time to "look for patterns — specifically, the most probably sentence/sentences that fit the question," nor does "probability" in terms of LLMs refer to "frequency — how many times something appears in its training data."
LLMs don't even have access to their training data at response generation time. And responses aren't created by "voting" on how many times it's seen a particular phrase (and it doesn't operate on sentences: it operates on word fragments, aka tokens).
I'd recommend Andrej Karpathy's "Neural Netowrks: Zero to Hero" YouTube lecture series (https://karpathy.ai/zero-to-hero.html), but here's a pretty condensed overview: the way LLMs work is they serially generate tokens when you ask them a question (generating tokens is referred to as "inference"). During training, we start with a random set of values for the model parameters (the "weights") and ask the model to predict the probabilities for token sequences. At the beginning of the process, it's usually very wrong, unless you won some unbelievably lucky universal dice roll. But we edit the values of the weights based on how right or wrong the model's predictions were by using backpropagation + a loss function to determine which parameters most influenced a particular prediction, and use gradient descent in that highly-dimensional space to perturb the parameters according to some value determined by an optimization function. Doing this zillions of times repeatedly is how we end up with the final values for the parameters — aka the model weights — it's not by "counting votes" or even "counting sequences," it's by calculating which parameter values are the best ones to predict the token probabilities. The frequency of tokens appearing in the dataset (or even the frequency of sentences) isn't directly computed, although extremely-high-frequency sequences might be memorized — but it can't memorize all of them, because the models are much smaller than the training set.
The theory for why this works so well is that since the models don't have enough space to memorize the entirety of the massive training set (i.e. Llama 3.1 8b is about 16GB, but was trained on the entire internet which is many orders of magnitude larger than that), the best values for the parameters are actually ones that create some sort of underlying world model for why that token sequence probability was predicted. That's very different than "counting votes," or counting sentence frequencies. Even if you disagree with this hypothesis for why the models work, you have to admit there's some underlying meaning being parsed out: it's not just memorization, even if it does sometimes memorize useful facts (useful, at least, to predict probabilities during training time). It simply can't memorize its training set: the training data is way too large compared to the size of the model, and it doesn't get to cheat and look at the training data when asked a question.
This idea is often used to argue that LLMs will never be capable of novel idea generation, but I don’t think it’s a good argument.
For one, the LLM has such a large breadth and depth of knowledge that it could conceivably learn relations between concepts in a way that no human has before.
Secondly, novel ideas occur at the margins. Very rare is the case where someone comes up with a fundamentally new idea out of the blue. Instead, novel ideas arise just at the edge of one’s expertise. If you dial up the temperature of an LLM, it will generate novelty, and then it’s just a matter of evaluating merit.
Iterated inference at the margins of LLM knowledge will lead to novel knowledge synthesis.