[tehnub]

That's why they require that you reach a certain threshold number of matches before its sent for human review. The threshold allows them to take the probability of a false collision, which they can estimate from data, and set the probability of an overall false-flag by requiring a certain number of these collisions. They've released that the threshold, to start, would have been 30 (Page 10 of https://www.apple.com/child-safety/pdf/Security_Threat_Model...). They claim that, given the probability of a false collision, and the threshold that they've set, the probability of your photos being sent for human review falsely is 1/trillion.

[fisherjeff]

They mention a “very conservative false positive rate” - doesn’t 1/trillion imply that they used 1 / (1e12 ^ (1/30)) = ~40% as the false positive rate? If so, that does seem extremely conservative to me!

[tehnub]

A 40% false collision probability would give an overall false flag probability of 1/trillion only if you had exactly 30 photos in your library, and thus all 30 had to be false collisions. The calculation gets a little more complicated if you have more, because you have to account for all the possibilities of combinations of 30+ false collisions among N photos, for N > 30. I wrote out the calculation in a comment from when this was being discussed a few months back: https://news.ycombinator.com/item?id=28174822.

On page 10 of the paper I linked though, they state that they assume a false collision probability of 1/million, which is more conservative than the 3 in 100 million false collisions they saw in their tests. The way they chose 30 as the threshold is based on the safeguarding assumption that everyone's photo library is larger than the actual largest library. This is safeguarding because the more photos you have, the more likely you are to have collisions. Copying from my previous comment, we can compute their photo library size assumption by solving for N in this equation: 1/trillion = 1 - sum_{k=0}^{29} of (N choose k) (1 - p)^k p^(N - k), where p is 1/million (the probability of a false collision).

[kfprt]

You are incorrectly assuming a non adversarial environment. Swatting 2.0.

[concinds]

The problem with this argument is that the "adversarial environment" argument applies to a worse degree to all cloud storage services who do the scanning in the cloud, since they have no threshold mechanism, and lack transparency on whether there is any human review whatsoever. You would still be reported to the police if someone hacks your Google Photos account and uploads CSAM to it.