[ben_w]

I agree it's more than a simple engineering challenge, but I do so because it is not entirely clear if even humans avoid this issue, or even if we merely minimise it.

We're full of seemingly weird cognitive biases: Roll a roulette wheel in front of people before asking them the percentage of African counties are in the UN, their answers correlate with the number on the wheel.

Most of us judge logical strengths of arguments by how believable the conclusion is; by repetition; by rhyme; and worse, knowledge of cognitive biases doesn't help as we tend to use that knowledge to dismiss conclusions we don't like rather than to test our own.

[Mawr]

How is that bias weird? It has a straightforward explanation - the visual system has an effect on reasoning. This, as well as other human biases, can be analyzed to understand their underlying causes, and consequently mitigated. LLM output has not discernible pattern to it, you cannot tell at all whether what it's saying is true or not.

[ben_w]

> How is that bias weird?

The people can see a random number that they know is random, and yet be influenced by it when attempting facts.

> LLM output has not discernible pattern to it, you cannot tell at all whether what it's saying is true or not.

LLMs are the pattern. This is a separate axis to "is it true?"