[stolsvik]

I thought this was obvious: They lack an “inner voice” (80% of humans?) or “inner imagery” (the rest) as we humans do, so they cannot first think the problem through before answering. Thus, using the actual “output area” as such a scratch pad can help it cover a larger area of reasoning before outputting an answer - just as we do.

I feel you can even see this when you ask it certain questions with “think in steps” prompting: It can output temporary thoughts which aren’t of use in the final answer - again just as we do when attacking a problem we can’t immediately answer.

Also, we humans often use pen and paper to jot down temporary and intermediary thoughts and answers. Again, LLMs don’t have that, but can use the output as something similar.

Some styles of ToT prompting actually make the LLM have two types of output - one for its “inner voice thinking”, and then another for output meant for the human. The same goes when one give the LLM method calling abilities, or “googling”: This can be seen as a way to perform thinking and reasoning without output meant for the user, before formulating an answer.