[viraptor]

> Doesn't an LLM pick the "most probable next symbol"

Yes, but that very rarely matters. (Almost never when it's brought up in discussions)

> Couldn't it then, if the probability falls below some threshold, say "I don't know" instead of giving what it knows is a low-probability answer?

A low probability doesn't necessarily mean something's incorrect. Responding to your question in French would also have very low probability, even if it's correct. There's also some nuance around what's classified as a hallucination... Maybe something in the training data did suggest that answer as correct.

There are ideas similar to this one though. It's just a bit more complex than pure probabilities going down. https://arxiv.org/abs/2405.19648

[anticensor]

> Responding to your question in French would also have very low probability, even if it's correct.

It's actually a common utterance in Paris.