| > What we call hallucination is usually when the model says something confidently wrong This is a poor definition that only applies to language models trained to be truthful. If you trained a language model to lie, and it told the truth, that would also be a hallucination. Or if a model was trained to never sound confident, and it made confident, but correct, claims. My definition is more accurate. > Yes the sampling procedure is nondeterministic but this is unrelated to hallucinations. It’s not the only factor, but it’s absolutely related. It’s also really easy to explain in a comment. For example, if you always sampled the lowest ranked token, the model would always hallucinate (by output mostly garbage) Top-k sampling doesn’t eliminate all errors, unless you’re just always picking the most likely token. At that point the sampling process is deterministic, but we’ve seen model output be poor with that setting for reasons I explain next. > that hallucination is a deeper problem Of course, it’s because the training process itself is nondeterministic. We can’t make a perfect model, it’s just not how statistical models work. |