[IshKebab]

I don't know how people can still be making this misguided comment. Predicting tokens well requires reasoning and planning.

"Brains are impressive and have their uses, but we shouldn't pretend like a muscle controller can somehow reason and plan."

[surgical_fire]

> Predicting tokens well requires reasoning and planning.

Reason and planning regarding token prediction

It cannot reason about the context of its output. It only infers the most likely token to follow up.

[IshKebab]

Of course it can reason about the context of its output. I'm honestly not sure what you're trying to say.

Can you give me an example of a prompt that requires your definition of "reasoning"?

[surgical_fire]

Have you ever tried to get whatever fancy LLM to write some code for you and had it generate code that was at the same time plausible to look at but complete bullshit?

This is what I am talking about. It can reason and plan for what is the likely token based off its training data. It is completely unable to evaluate the logic of the code it was generating. It can not reason on the context (in this case, programming).

The same is true for other domains. Law, Medicine, etc. the stricter the field, the less reliable LLMs are, because it cannot reason about the context of what it is writing.

That said, I like LLMs, and I think it was an interesting productivity tool, only having been hyped beyond any reasonable expectations. I find it more useful for less strict contexts (for example, creative writing).

[marvinkennis]

I mean obviously predicting tokens in the context of LLM output requires planning. But planning tokens doesn’t generalize. This is evident when you train an LLM on a small data set and ask it about any unseen variation of it.

There are many examples of slightly modified popular riddles that are easy to solve by reasoning about them, but LLMs always fall back to the most likely output from their training data.