[marvinkennis]

I mean obviously predicting tokens in the context of LLM output requires planning. But planning tokens doesn’t generalize. This is evident when you train an LLM on a small data set and ask it about any unseen variation of it.

There are many examples of slightly modified popular riddles that are easy to solve by reasoning about them, but LLMs always fall back to the most likely output from their training data.