[skybrian]

The linked paper is about detecting when the LLM is choosing randomly versus consistently at the level of factoids. Procedurally-generated randomness can be great for some things like brainstorming, while consistency suggests that it's repeating something that also appeared fairly consistently in the training material. So it might be true or false, but it's more likely to have gotten it from somewhere.

Knowing how random the information is seems like a small step forward.

[caseyy]

I don’t know. It could be a misleading step.

Take social media like Reddit for example. It has a filtering mechanism for content that elevates low-entropy thoughts people commonly express and agree with. And I don’t think that necessarily equates such popular ideas there to the truth.

[skybrian]

The conversations about people being misled by LLM's remind me of when the Internet was new (not safe!), when Wikipedia was new (not safe!) and social media was new (still not safe!)

And they're right, it's not safe! Yes, people will certainly be misled. The Internet is not safe for gullible people, and LLM's are very gullible too.

With some work, eventually they might get LLM's to be about as accurate as Wikipedia. People will likely trust it too much, but the same is true of Wikipedia.

I think it's best to treat LLM's as a fairly accurate hint provider. A source of good hints can be a very useful component of a larger system, if there's something else doing the vetting.

But if you want to know whether something is true, you need some other way of checking it. An LLM cannot check anything for you - that's up to you. If you have no way of checking its hints, you're in trouble.