[TZubiri]

You might be in the wrong thread. This is merely a comment about whether LLMs hold a concept of uncertainty, they do.

Also, the next token might be 2 and the next token might be ², the next token could also have been x, these are all valid statements and the LLM might have been uncertain because of them.

2+2=4

2+2=x

2+2=2x

2+2=2x2

2+2=2²

Are all valid statements.

[glandium]

And somewhere in its training data, you can be sure there's also 2+2=5.

[TZubiri]

Yes, but most likely it's marked as false or incorrect through fine tuning or some form of reinforcement.

The idea that the logprobs of any token is proportional to the amount of times it comes up in training data is not true.

For example, suppose that A is a common misconception and is repeated often in Reddit, but B appears in scholarly textbooks and papers, and higher reputation data sources. Then through reinforcement the logprobs of B can increase, and they can increase consistently when surrounded by contexts like "This is true" and conversely decrease in contexts of "this is not true".

So the presumptions and values of its trainers are also embedded into the LLM in addition to those of the authors of the text corpus.