[inglor_cz]

With some humans, you can at least rely on their humility and ability to say "I don't know". This is a positive trait in people and I would rely on such honest people much more than on anyone who has all the answers to everything.

The machine seems to be unable to say or even detect that it does not know. At the same time, it communicates in flawless English (or whatever the current setting is), which is a trait we tend to associate with highly educated people from the real world. This short-circuits our bullshit detectors a bit.

[ben_w]

> With some humans, you can at least rely on their humility and ability to say "I don't know". This is a positive trait in people and I would rely on such honest people much more than on anyone who has all the answers to everything.

You might, and I try to. Humanity as a whole? In practice, highly confident people who are totally sure but wrong, still get listened to over people who are humble and aware of their limits.

Humans also short-circuit each other's BS detectors.

[K0balt]

The bias to assume that computers are going to produce correct answers is extremely strong.

People intuit that Wikipedia is written by people, so they can apply that knowledge appropriately.

For some reason, most people have a knee jerk reaction to a fully synthetic statement that biases them strongly towards the assumption of veracity.

I always think of LLMs as “my functioning alcoholic veteran friend bob, who has several PHDs and was blown up a couple of times in Iraq”. That seems to be a good framework in order to intuit the usefulness of llm generated output.

[inglor_cz]

"The bias to assume that computers are going to produce correct answers is extremely strong."

This. We know that computers are very good at actual computation, and we don't expect them to go completely haywire in conversations either.

Though this is beginning to change, with the observation of just how blatant some of the hallucations are, accusing random people of serious crimes etc. But the pro-computer bias is still strong.

There was an awful case of a system in the UK which accused postal officers of defraudation. The software malfunctioned, but people were indicted and punished by the courts relying on infallibility of computers, and some of the innocent victims committed suicide out of shame.