[miki123211]

There's one thing I wish people understood about LLMs, and it doesn't really have anything to do with what's inside the neural network part. It's the fact that LLMs can only write in one direction — forward.

When you are writing an essay and realize midway through a sentence that what you've written doesn't make sense, you go back and edit. An LLM can't do that, the only thing it can do is keep on generating. Because training data typically contains full essays and not half-finished sentences which were then edited, LLMs have a strong preference for "saving face" and producing grammatically correct, internally coherent outputs. They will often do so even if the only way to write themselves out of the corner they wrote themselves into is to lie. To maintain internal coherence, they'll then repeat that lie for the rest of the response.

This is also why changing response structure used to affect LLM performance so dramatically. If you asked an LLM to solve a math problem and all-but-forced it to start with the answer, it would have had to calculate that answer before emitting any tokens, something which it very often wasn't able to do. If it was told to follow up the answer with an explanation, it would produce a plausible-sounding explanation to maintain coherence.

If, on the other hand, it was told to start by "thinking step by step", it would often be able to solve the first step, and then the next one given the results of the first, and so on, until it was able to reach the answer. Because the answer came last, it wasn't committing to anything, so had no reason to "save face" and lie.

This part of the problem is basically solved now with reasoning; reasoning is where all the step-by-step stuff happens, even if users aren't always able to see it. In the process of RLVR, models even train themselves into outputting phrases like "let me check my answer once again" in the chain-of-thought; those serve as their "life rafts" which they can use to both save face and change their answer.

[swyx]

agree reasoning fixed a lot of it. but the inherent path dependence of autoregression is one reason i was excited for text diffusion models (https://www.youtube.com/watch?v=r305-aQTaU0)

instead of going left to right, even with a scratchpad, maybe you start with a rough shape of the big picture all at once, and then you iteratively resolve and things come into focus.

mercury (https://www.youtube.com/watch?v=2fDBeMu6xjk) seems to have made the most progress here, which is not saying a ton but is not nothing. i do think it is telling that of the big labs, only GDM has made any meaningful bet on text diffusion. you can bet your ass all of them have evaluated it for a source of alpha.

[chris_money202]

In terms of our brains though we can only think forward as well (if forward is time). Our brain in the future says something we did in the past was wrong (part of the sentence we wrote) and that informs our body (the agent) to go back and fix it

[kzrdude]

I think that's why pen and paper is such a good tool for thinking. :)

[FergusArgyll]

I get what you're saying but to be slightly pedantic etc.

Why can't an llm tool call `delete(index: int)` or `replace(from_index: int, to_index: int, string: str)` and then it can go back and edit just the way we can?

We also first made the mistake and only afterwards noticed we actually want to change something

[podocarp]

It can, that's why things like planning and reasoning are so effective. Planning effectively gives them a scratchpad they can keep modifying till they're satisfied without being constrained into the "only go forward" output they usually produce. Reasoning is also the same idea. Although I have on rare occasions seen in plans models saying "blabla above is wrong, I should instead do …" instead of actually fixing the plan, but that's another problem