[voidhorse]

The whole thing is silly. Look, we know that LLMs are just really good word predictors. Any argument that they are thinking is essentially predicated on marketing materials that embrace anthropomorphic metaphors to an extreme degree.

Is it possible that reason could emerge as the byproduct of being really good at predicting words? Maybe, but this depends on the antecedent claim that much if not all of reason is strictly representational and strictly linguistic. It's not obvious to me that this is the case. Many people think in images as direct sense datum, and it's not clear that a digital representation of this is equivalent to the thing in itself.

To use an example another HN'er suggested, We don't claim that submarines are swimming. Why are we so quick to claim that LLMs are "reasoning"?

[Velorivox]

> Is it possible that reason could emerge as the byproduct of being really good at predicting words?

Imagine we had such marketing behind wheels — they move, so they must be like legs on the inside. Then we run around imagining what the blood vessels and bones must look like inside the wheel. Nevermind that neither the structure nor the procedure has anything to do with legs whatsoever.

Sadly, whoever named it artificial intelligence and neural networks likely knew exactly what they were doing.

[SubiculumCode]

I was having a discussion with Gemini. It claimed that because Gemini, as a large language model, cannot experience emotion, that the output of Gemini is less likely to be emotionally motivated. I countered that the experience of emotion is irrelevant. Gemini was trained on data written by humans who do experience emotion, who often wrote to express that emotion, and thus Gemini's output can be emotionally motivated, by proxy.

[rented_mule]

> this depends on the antecedent claim that much if not all of reason is strictly representational and strictly linguistic. It's not obvious to me that this is the case

I'm with you on this. Software engineers talk about being in the flow when they are at their most productive. For me, the telltale sign of being in the flow is that I'm no longer thinking in English, but I'm somehow navigating the problem / solution space more intuitively. The same thing happens in many other domains. We learn to walk long before we have the language for all the cognitive processes required. I don't think we deeply understand what's going in these situations, so how are we going to build something to emulate it? I certainly don't consciously predict the next token, especially when I'm in the flow.

And why would we try to emulate how we do it? I'd much rather have technology that complements. I want different failure modes and different abilities so that we can achieve more with these tools than we could by just adding subservient humans. The good news is that everything we've built so far is succeeding at this!

We'll know that society is finally starting to understand these technologies and how to apply them when we are able to get away from using science fiction tropes to talk about them. The people I know who develop LLMs for a living, and the others I know that are creating the most interesting applications of them, already talk about them as tools without any need to anthropomorphize. It's sad to watch their frustration as they are slowed down every time a person in power shows up with a vision based on assumptions of human-like qualities rather than a vision informed by the actual qualities of the technology.

Maybe I'm being too harsh or impatient? I suppose we had to slowly come to understand the unique qualities of a "car" before we could stop limiting our thinking by referring to it as a "horseless carriage".

[voidhorse]

Couldn't agree more. I look forward to the other side of this current craze where we actually have reasonable language around what these machines are best for.

On a more general level, I also never understood this urge to build machines that are "just like us". Like you I want machines that, arguably, are best characterized by the ways in which they are not like us—more reliable, more precise, serving a specific function. It's telling that critiques of the failures of LLMs are often met with "humans have the same problems"—why are humans the bar? We have plenty of humans. We don't need more humans. If we're investing so much time and energy, shouldn't the bar be bette than humans? And if it isn't, why isn't it? Oh, right it's because actually human error is good enough and the actual benefit of these tools is that they are humans that can work without break, don't have autonomy, and that you don't need to listen to or pay. The main beneficiaries of this path are capital owners who just want free labor. That's literally all this is. People who actually want to build stuff want precision machines that are tailored for the task at hand, not some grab bag of sort of works sometimes stochastic doohickeys.

[trainerxr50]

I think more importantly there is this stupid argument that because the submarine is not swimming it will never be able to "swim" as fast as us.

This is true of course in a pointlessly rhetorical sense.

Completely absurd though once we change "swimming" to the more precise "moving through water".

The solution is not to put arms and legs on the submarine so it can ACTUALLY swim.

It would be quite trivial to make a Gary Marcus style argument that humans still can't fly. We would need much longer and wider arms, much less core body mass, feathers.

[cageface]

but this depends on the antecedent claim that much if not all of reason is strictly representational and strictly linguistic.

Most of these newer models are multi-modal, so tokens aren't necessary linguistic.

[comp_throw7]

What use of the word "reasoning" are you trying to claim that current language models knowably fail to qualify for, except that it wasn't done by a human?

[sgt101]

Well - all of them.

The mechanism by which they work prohibits reasoning.

This is easy to see if you look at a transformer architecture and think through what each step is doing.

The amazing thing is that they produce coherent speech, but they literally can't reason.

[comp_throw7]

This feels like we're playing word games which don't actually let us make useful claims about reality or predictions about the future. If we're talking purely about the model internals, without reference to their outputs, then your claim is wrong because we don't have a good enough understanding of the model internals to confidently rule out most possibilities. (I'm familiar with the transformer architecture; indeed this is why I asked what definition of the word reasoning the OP cared about. Nothing about transformers as an architecture for _training model weights_ prohibits the resulting model weights from containing algorithms that we would call "reasoning" if we understood them properly.) If we're talking about outputs, then it's definitely wrong, unless you are determined to rule out most things that people would call reasoning when done by humans.

[sgt101]

I might be able to learn more by chatting with you.

I think that the trained transformer has fixed weights and therefore cannot learn.

I think learning is one aspect of reasoning, and is demonstrated by challenges like navigation or puzzle solving where learning that one route to a solution is impossible is important.

I also think that the single forward pass of the model means that cyclic reasoning isn't feasible and that conditioning output by asking the model to "think" even when that thinking is done on the single forward pass means that logical processes are ruled out. The model isn't thinking in that case, the probabilities of the final part of the output are conditioned by requiring a longer initial output.

[etaioinshrdlu]

I don't think it's accurate anymore to say LLMs are just really good word predictors. Especially in the last year, they are trained with reinforcement learning to solve specific problems. They are functions that predict next tokens, but the function they are trained to approximate doesn't have to be just plain internet text.

[voidhorse]

Yeah, that's fair. It's probably more accurate to call them sequence predictors or general data predictors than to limit it to words (unless we mean words in the broad, mathematical sense) they are free monoid emulators

[antonvs]

And what are humans?

[sgt101]

Humans are humans - to deny that we are thinking, reasoning, living beings is a strange thing to do.

You can taste a beer, laugh so much it hurts, come to know how something works.

[antonvs]

I didn’t deny anything.

The parent comments were attempting to characterize LLMs as something more general than “word predictors”. The alternative “sequence predictors” was proposed.

My question relates to whether we have any reason to believe that the relevant aspects of human cognition are anything more than that.

Certainly humans have some advantages, like the ability to continuously learn (although there’s very strong evidence that we have a pretraining phase too, for example the difficulty of learning new languages as an adult vs. as a child.) But fundamentally, it’s not clear to me that our own language production skills aren’t “just” sequence prediction.

Perhaps, as the OP article speculates, there are other important components, like “models of the world”. But in that case, it may be that we’re augmented sequence predictors.