[cmsj]

You are absolutely correct that neither automatic nor manual review is ever going to be 100% accurate.

I would like to believe though that for this system to fully fail an innocent person, the following would all need to have failed:

1) Coincidental CSAM hash collision 2) Incorrect manual review by Apple 3) Incorrect subsequent review by NCMEC 4) Inability of a lawyer to obtain the original image for presentation during a trial/appeal

which seems kind of unlikely? (although it's certainly the case that once steps 1, 2 and 3 have failed, the person's reputation is likely damaged even if they are able to prove their innocence in court).

The wider question here is, should 100% accuracy be the bar by which we judge this? I don't think we expect the law enforcement system to be 100% right, hence principles like the presumption of innocence and right to appeal, and even then it gets things wrong sometimes.

[alpaca128]

There are known cases of police faking AI-generated evidence[0]. There's no reason why Apple would be immune against such things. And the recent British post office scandal shows that even without manipulation false faith in technology as evidence can destroy hundreds of lives. The low chance of an error going through that whole chain of checks also increases the trust in that system even in the case of a false positive.

And all this is assuming it will never be expanded from CSAM to other content. Apple is already rolling out a censored version of iOS in China.

[0] https://www.vice.com/en/article/qj8xbq/police-are-telling-sh...

[samrolken]

You’re missing the threshold that is part of this system. You would need multiple hash collisions across multiple photos to trigger these mechanisms.