[claytonjy]

Is there a longer-form paper on this yet? TPR (P(T|AI)) and FPR (P(T|H)) are useful, but what I really want is the probability that a piece flagged as AI-generated is indeed AI-generated, i.e. P(AI|T). Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI.

If we assume the challenger set is evenly split 50-50, that means

    P(AI|T) = P(T|AI)P(AI)/P(T) = (0.26)(0.5)/(0.26+0.09) ~ 37%

So slightly better than a 1/3 chance of the flagged text actually being AI-generated.

They say the web-app uses a confidence threshold to keep the FPR low, so maybe these numbers get a bit better, but very far from being used as a detector anywhere it matters.

[TchoBeer]

>Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI

This will obviously depend on your circumstances.

[drc500free]

Precision is impossible to calculate without knowing P(AI), which is use-case specific.

Source: Spent 10 years trying to explain this to government people who insisted that someone tell them Precision based purely on the classifier accuracy without considering usage.