[claytongulick]

> LLMs do reason

No, they don't.

They are token predictors that use statistical techniques to emit the randomly weighted next most likely token given the previous token list.

The result is a strange mimic of human reasoning, because the tokens it predicts are trained on strings that were produced by humans that were reasoning, but that's not the same thing.

Human cognition is complex and poorly understood, and the nature of the mind is an area of study almost as old as consciousness itself. We don't know exactly how it works, or what its exact relationship to the brain is, but we do know that it is not a simple token predictor.

LLMs, by their very nature are constrained to the concept of language and the relationship between existing words in a corpus. This is a box they can not escape.

Modern neuroscience suggests that the human brain is much more vast than that, and in many ways looks like it is constrained by language, but certainly not limited to it.

[antonvs]

> They are token predictors that use statistical techniques to emit the randomly weighted next most likely token given the previous token list.

Sounds like an implementation detail. Now describe how human reasoning works and explain why that process of chemical and electrical signals results in "reasoning" whereas what LLMs do isn't.

The problem with being this reductive is you can do it to anything, including humans. You can’t be reductive about LLMs and refuse to be reductive about humans - that's poor reasoning, and an LLM would out-reason you on this point, further negating your case.

[claytongulick]

Human cognition is poorly understood and much more complex than it seems.

For an example, look at some of Julia Mossbridge's work.

If even a small part of her work is true and valid, it points to something far outside our current framework.

You don't need to go as far afield as Mossbridge, though - that's an extreme example. Pretty much any modern neuroscience will make you question a lot of assumptions, at least it did for me.

[CrazyStat]

> For an example, look at some of Julia Mossbridge's work.

Never heard of her but I just spent about 5 minutes looking.

Her PhD is in communication sciences and disorders [1], but apparently she’s a quantum physicist now:

> AMELIA is built on the Causally Ambiguous Duration-Sorting (CADS) effect — a breakthrough discovery by Dr. Julia Mossbridge showing that light, under classical boundary conditions, behaves differently based on future temporal boundaries. [2]

Filed under crank, not going to bother investigating further.

[1] https://books.google.com/books/about/Have_a_Nice_Disclosure....

[2] https://americanelectrodynamics.com/#technology

[dnautics]

You have moved goalposts from reasoning to "human cognition". I won't tolerate that sort of slippery wordplay.

Reasoning is making analogies between logical patterns found in conceptual space, with a direction of time (statements precede conclusions). For example. A => B and B => C. You may now deduce A => C. For something fuzzier, A~D and B~E, you may now deduce that D~=>E. This is the sort of thing that higher layer attention mechanism is capable of doing.

> This is a box they can not escape.

Would you say that Helen Keller was less capable of abstract reasoning because she had more constrained access to sensory input?

[claytongulick]

Reasoning requires cognition, otherwise there's nothing to reason about, no context or value system to use as a basis for reason.

Decision making can be done by trained machines following rules, but that's different that reasoning. A thermostat isn't reasoning when it decides to turn on the air conditioner, to argue otherwise expands the definition of "reason" to be so broad that it becomes useless.

LLMs are trained on human knowledge and reasoning that results from human cognition, and they are excellent at stochastic mimicry - if the argument is that they are actually reasoning, then some sort of equivalent to human cognition must be present for that to be true. Lacking that, they are nothing more than "token extrusion machines" with some potentially useful characteristics.

[digitaltrees]

Why does reasoning require cognition? Isn’t a if else block or switch statement reasoning? Or a formal logic proof? If an LLM produces an output using formal logic or a python script why is that not reasoning? A human would offload the reasoning using similar methods. I know when I took the LSAT, I learned ways to diagram arguments and didn’t have to think/reason about it because the formal logic diagram did the “reasoning for me”.

Aren’t humans just “action potential” extrusion machines? What is unique about our neural pattern recognition to make our cognition different in nature rather than merely degree?

It seems clear at this point that the greatest insight that unlocked our current AI acceleration was scaling alone would unlock emergent properties and abilities.

[dnautics]

"acceleration was scaling alone would unlock emergent properties and abilities."

Agreed but I would frame it in the negative, "don't worry about overfitting, the lucky ticket hypothesis just works "

[Jtarii]

Can you give a concrete example of something that is impossible for an LLM to ever do due to its lack of reasoning ability.

[claytongulick]

I try to avoid absolutes, "ever" is a long time. Who knows, maybe we crack the code of cognition at some point?

In the meantime, these [1] are pretty funny.

[1] https://x.com/huskirl

[NobleLie]

The problem with that is LLMs can output words or symbols that seen like it used "reason" to produce. But for everything the core algorithm does, it's simply nothing like the wetware reasoning to get to the same answer. So he didn't move goalposts. He always meant the reasoning that stems from human cognition.

Technically if it has that, it'd be singularity no? So basically the premise is they are doing nothing of the sort. Prove any LLM enough and it really does show it has no quarrels contradicting itself or being bossed around. Has no belief / no orientation etc. It's truly mindless but tricks our mind and soul (or whatever) probably.

[dnautics]

> Technically if it has that, it'd be singularity no?

reasoning is not black and white. It is possible to reason poorly. Most people cannot do basic math proofs, even math majors struggle with the hardest math proofs. Reasoning in humans is also context/token dependent. I just spent one HOUR trying to show my mom (who has mild dementia) how to use amazon fire (push DOWN until your channel shows up, push RIGHT until the channel becomes big) and she could not figure it out. Rewrote the instructions in japanese and she followed the logic relatively smoothly. Ironically, i'm pretty sure her english is better than her japanese, vocabulary wise.

> it's simply nothing like the wetware reasoning to get to the same answer.

but you don't know how wetware reasoning works, so you are incapable of making that proclamation. I'm pretty sure when I do math proofs (I'm not an amazing mathematician) sometimes I have to literally tick my way through each step of the proof, sometimes breaking it down to super-basic substeps, which to me feels awful lot like what an LLM could be doing. For that matter we don't know how LLM reasoning works but my claim is that these LLMs are in principle capable of reasoning due to architecture.

If this doesn't make sense I suggest you look over the architecture of LLMs carefully and try to understand my point.

(BTW I'm not talking about "reasoning models" with "thinking turns", that's just marketing speak, I'm talking about ANY transformer-based model, even the "dumbest UX architecture" completion models)

[digitaltrees]

Humans off load reasoning into language and syntax. Chinese encodes arithmetic into the grammar/syntax patterns better than French for example.

Your posts are generally insightful. Thanks for the contribution. Even if it’s a bit cranky and gruff :)

[digitaltrees]

The structure of language encodes logic in many ways. So the models ability to reason may be an emergent property of the reasoning ability humanity has ejected an extracted from our neural networks and abstracted into language a symbols.

[keeganpoppen]

there is absolutely no line of demarcation between human reasoning and what you described