[kostaj]

Agree that some of the claims are forward-looking. The messiness of the real-world and real-user fact checks. No ground-truth verdicts are provided or used in the study though. It only measures the level of agreement between the selected models, not which one is right on which claim. I.e. none of the claims is actually labelled.

[brokensegue]

were you involved in making the study? your bio says you work for them so you should probably indicate that in your comments.

lack of agreement when there is no singular correct answer (or any answer at all) isn't a useful metric

I ran into a lot of these kinds of issues when working on the Citation Needed WMF project (and related extensions). Truth is so often very nuanced.

[simonw]

They introduced themselves as the study author here: https://news.ycombinator.com/item?id=48307887#48307899

[brokensegue]

ah. I missed that.