[proofofcontempt]

What does this show that we didn't know already? LLMs cannot provide accurate answers to questions where data is not included in their training sets. This doesn't appear to have much substance

[dragandj]

LLMs can and will provide inaccurate answers to questions where data is included in their training sets too, that's in the nature of neural networks. It's just less likely that when the data is not in the training set...

[101008]

Unfortunately most people are not aware of this and treat LLM models as this superpowered brain who knows everything and can do everything.

[dncornholio]

They will happily google it for you and give you the top reddit comment.

This is worse.

[zug_zug]

Well then it shows that these models are using widely disparate training sets and have high confidence even when they shouldn't.

Questions like "is mouthwash effective" presumably has one solid data source -- medical journals.

[simonw]

But the prompt didn't give the models the option to say "I don't know", so it wasn't a measure of their confidence.

[zug_zug]

I mean that's true but I don't think that's realistically what's going on when one model gives an unqualified "Yes" and the other gives an unqualified "no."

You can argue the study isn't as case-closed-decisive as we'd ideally like, but it's certainly evidence. It's probably hard to design a better study.

[TaupeRanger]

What are you talking about? The models were not ALLOWED to have confidence (or the lack thereof). They were explicitly told to give a single label, and in most cases, all of them were correct depending on additional context they would surely have provided, especially with access to the internet (which some didn't have). This is just silly.