[dwa3592]

For non-chess players - imagine if Geoffrey hinton used AI detectors (GPT zero, pangram etc) to accuse and punish people of using AI in their writing and bad mouth them publicly. That's what kramnik (former world chess champion) did to Danya and other players. I miss Danya.

[wilbur_whateley]

How did he punish them?

[nopinsight]

To my knowledge, Pangram has a very low false positive rate approaching zero. (False negative is another matter.) I’m not sure that’s what you want to use as the analogy here? (I don’t know much about this Kramnik situation.)

[TimPC]

Pangram’s market claims are very different from independent validations of the platform. The 99.98% rate is on a toy dataset not resembling reality.

Even if 2/10000 is true that’s nowhere near accurate enough to make aggressive accusations that create anxiety at levels people need to medicate with potentially fatal consequences.

[nopinsight]

Someone also input lots of pre-GPT texts to Pangram and they were all clear. The point is Kramnik’s claims are much weaker and not comparable.

[Seattle3503]

> Pangram’s market claims are very different from independent validations of the platform.

Do you have a link to independent validations? The ones I found, like an upcoming nber paper confirm the companies claims.

[dwa3592]

https://github.com/deepanwadhwa/ai_detector_fails/blob/main/...

the text in above is all ai generated. i uploaded a picture of two kittens to gemini and asked it to guess their age. then asked gemini to change the language to of someone who is born in deep Appalachia but has Japanese parents. then asked it to change the bullets to paragraphs written by someone who is talking fro life experience rather than grounded in published science. pangram gave it 100% human generated. go ahead, try it. it boggles me how people believe these ai detectors.

if it makes you feel good the original response from gemini gets 100% AI detection and for that I think pangram should get the credit but in practical terms the empty pots in my yard are more useful than pangram at this point.

[Seattle3503]

You are talking about false negatives, but this thread started with a discussion around the potential for false positives causing writers anxiety

> Even if 2/10000 is true that’s nowhere near accurate enough to make aggressive accusations that create anxiety at levels people need to medicate with potentially fatal consequences.

False positives and false negatives are different problems that have different impacts.

Does your single example prove that Pangram doesn't work, or that it doesn't work on short snippets of text? Try to get a detection error on a longer run of text. I'd be curious to see the results, particularly if you can get a false positive.

[dwa3592]

Well, false positives must anyway be zero for a product which is built around detecting AI. let me explain - the value proposition of this product is that we detect AI. Its not we don't call human writing AI generated. The latter is implied and must not happen ever. The value is that pangram detects AI generated content which it did not in the above example. if this explanation doesn't make sense then think of pangram as a pregnancy test where presence of AI == presence of baby. If a pregnancy test keeps saying you are not pregnant when you actually are then that's a problem, right? - you can't argue that at least its not saying you are pregnant when you are not.

But here's a bigger problem - Imagine a teacher grading 50 students - 40 of them use AI to write their answers and 10 write honestly. All 40 of them use the hack that I used and get 100% human from pangram and the other 10 also get 100 human from pangram. what is the product really adding to the workflow? Nothing, zero or zilch!!! - but then you come along and say hey at least those 10 honest students also got 100% from pangram. This whole argument of false positives being very very low works if your false negatives are tight which they are not.

[dwa3592]

>>To my knowledge, pangram has a very low false positive rate approaching zero. (False negative is another matter.) I’m not sure that’s what you want to use as the analogy here?

'To my knowledge' - okay. In my experience, they're all horrible. I have fooled them both ways - ai generated content that they say is purely human and human generated content that they say is partially AI or just AI.