Hacker News new | ask | show | jobs
by andai 31 days ago
Didn't Pangram claim to have a >99% success rate?

https://news.ycombinator.com/item?id=48667761

From their homepage:

> Detect AI-generated content with 99.98% accuracy.

Are they wrong?

Though I ran the numbers, and even with a 0.02% false positive rate, that works out to about 6000 students falsely accused every semester, per university.
2 comments
Chu4eeno 31 days ago
It's also confirmed in third-party studies like this https://arxiv.org/html/2501.15654v1 (which also confirms that humans that have experience with LLMs are very good at spotting them).
link
lelanthran 31 days ago
> > Detect AI-generated content with 99.98% accuracy.

> Though I ran the numbers, and even with a 0.02% false positive rate

They don't say that the false positive is 0.02%, only that the accuracy is 0.02%. All we know for certain is that the false positive and false negatives added together result in 0.02%.

link
andai 30 days ago
Yeah, my bad. Well they don't exactly guard against that interpretation. "Up to 6000 falsely accused per semester!" doesn't exactly ease the mind!
link