[dartos]

> while we literally watch reasoning models say things like "oh that's not right, let me try a different approach".

Not saying I disagree with your premise that errors can’t be corrected by using more and more tokens, but this argument is weird to me.

The model isn’t intentionally generating text. The kinds of “oh let me try a different approach” lines I see are often followed by the same approach just taken. I wouldn’t say most of the time, but often enough that I notice.

Just because a model generates text doesn’t mean that the text actually represents anything at all, let alone a reflection of an internal process.

[TeMPOraL]

> Just because a model generates text doesn’t mean that the text actually represents anything at all, let alone a reflection of an internal process.

What does it represent then? What are all these billion weights for? It's not a bag full of NULLs that just pulls next words from a look-up table. Obviously there is some kind of internal process.

Also I don't get why people ignore the temporal aspect. Humans too generate thoughts in sequence, and can't arbitrarily mutate what came before. Time and memory is what forces sequential order - we too just keep piling on more thoughts to correct previous thoughts while they are still in working memory (context).

[_heimdall]

The text represents a prediction of how a human may respond, one word(ish) at a time, that's it.

With "reasoning" models, the reasoning layer is basically another LLM instructed to specifically predict how a human may respond to the underlying LLM's answer, fake prompt engineering if you will.

There of course is some kind of internal process, but we can't prove any kind of reasoning. We ask a question, the main LLM responds, and we see how the reasoning layer LLM itself responds to that.

[_0ffh]

Please don't confuse people with wrong information, the reasoning part in reasoning models is the exact same LLM that produces the final answer. For example o1 uses special "thinking" tokens to demarcate between reasoning and answer sections of it's output.

[_heimdall]

Sure, that's a great clarification though maybr a bit of an implementation detail in this context.

Functionally my argument stands in this context - just because we can see one stream of LLM responses responding to the primary response stream says nothing of reasoning or what is going on internally in the reasoning layer.

[dartos]

> what is going on internally in the reasoning layer.

We literally know exactly what is going on with every layer.

It’s well defined. There are mathematical proofs for everything.

Moreover it’s all machine instructions which can be observed.

The emergent properties we see in LLMs are surprising and impressive, but not magic. Internally what is happening is a bunch of matrix multiplications.

There’s no internal thought or process or anything like that.

It’s all “just” math.

To assume anything else is personification bias.

To look at LLMs outputting text and a human writing text and think “oh these two things must be working in the same way” is just… not a very critical line of thought.

[_heimdall]

> We literally know exactly what is going on with every layer.

Unless I missed a huge break in the observability problem, this isn't correct.

We know exactly how every layer is designed and we know how we functionally expect that to work. We don't know what actually happens in the model at time of inference.

I.e. we know what pieces were used to build the thing but when we actually use it its a black box - we only know inputs and outputs.

[nyrikki]

Subbarao Kambhampati, who seems to only use X is a good resource. He points out how the CoT text is not of semantic importantce.

This work from his team shows how few 'reasoning' traces are valid.

https://atharva.gundawar.com/searchformer_response_analysis....

This paper shows how the scratch space gets transformers to PTIME from TC0 without it.

https://arxiv.org/abs/2502.02393

OpenAI may be able to do more in the long term because they don't show the <think> and can spend more of that scratch space on improving answers vs appeasing users, but time will show.

Remember that probabilistic checkable proofs show how random data can improve computation.

The AI field has always had a problem with wishful mnomics.

But it is probably not a binary choice, if we could get the scratch space to reliably simulate Dykstra' shunting and convert to postfix as an example, that would be great.

[dartos]

> Humans too generate thoughts in sequence,

You don’t know this. I don’t feel like I generate thoughts in sequence, for me it feels hierarchical.

> can't arbitrarily mutate what came before

Uhh… what?

Do you remember your memories as a child? Or what you ate for breakfast 3 weeks ago?

Have you ever misremembered an event or half remembered a solution to a problem?

The information in human minds are entirely mutable. They are not like computers…

> It's not a bag full of NULLs that just pulls next words from a look-up table.

Funny enough, the attention mechanism that’s popular right now is effectively lots and lots of stacked look up tables. That’s how it’s taught as well (what with the Q K and V)

Tho I don’t think that’s a requirement for LLMs in general.

I find a lot of people who half understand cognition and understand computing look at LLMs and work backwards to convince themselves that it’s “thinking” or doing more cognitive functions like we humans do. It’s personification bias.

[Tostino]

Not OP.

> Do you remember your memories as a child? Or what you ate for breakfast 3 weeks ago?

For me, this seems like conjuring up and thinking about a childhood event is like putting what came out of my nebulous 'memory' fresh into context at the point in time you are thinking about it, along with whatever thoughts I had about it (how embarrassed I was, how I felt proud because of X, etc). As that context fades into the past, some of those thoughts may get mixed back into that region of my 'memory' associated with that event.

[naasking]

> The model isn’t intentionally generating text.

What's the mechanistic model of "intention" that you're using to claim that there is no intention in the model's operation?

> Just because a model generates text doesn’t mean that the text actually represents anything at all, let alone a reflection of an internal process.

Generating text is the trace of an internal process in an LLM.

[dartos]

> What's the mechanistic model of "intention" that you're using to claim that there is no intention in the model's operation?

You can’t prove intention, but I can show examples of LLMs lacking intent (as when repeating the same solution even after being told it was incorrect)

> Generating text is the trace of an internal process in an LLM.

Not really sure precisely what you mean by trace, but the output from an LLM (as with any statistical model) is the result of the calculations, not a representation of some emergent internal state.

[naasking]

> You can’t prove intention, but I can show examples of LLMs lacking intent (as when repeating the same solution even after being told it was incorrect)

I don't think that shows lack of intent, any more than someone who has dementia forgetting why they entered a room shows they lack intent.