[mitthrowaway2]

> LLMs produce their answers with a fixed amount of computation per token

I'm not that confident that humans don't do this. Neurons are slow enough that we can't really have a very large number of sequential steps behind a given thought. Longer complex considerations are difficult (for me at least) without at least thinking out loud to cache my thoughts in audible memory, or having a piece of paper to store and review my reasoning steps. I'm not sure this is very different than a LLM prompted to reason step by step.

The main difference I can think of is that humans can learn, while LLMs have fixed weights after training. For example, once I've thought carefully and convinced myself through step-by-step reasoning, I'll remember that conclusion and fit it into my knowledge framework, potentially re-evaluating other beliefs. That's something today's LLMs don't do, but mainly for practical reasons, rather than theoretical ones.

I believe the extent of world modelling done by LLMs still remains an open question.

[aeternum]

Yes, this is key. This idea that humans also require sequences to think was popularized by Jeff Hawkins even before all the LLM hype.

He was able to show that the equivalent of place cells (normally used to determine one's physical location) fire sequentially when humans perform tasks like listening to music or imagine feeling along the rim of a coffee cup.

The think step-by-step trick might just be scratching the surface of the various mechanisms we can use to give LLMs this kind of internal voice.