[onepunchedman]

Most people wouldn't of course. In this scenario you'd get someone to download the CSAM unknowingly. If they have iCloud sync it automatically uploads to iCloud, thereby triggering the system. At that point the authorities will be alerted by Apple, and you can inform media outlets. They in turn will ask law enforcement who will confirm the investigation, and the reputation of the person investigated will be tarnished.

[onepunchedman]

Also as dannyw pointed out, you don't even have to send CSAM to trigger the system. If they found you you would still be charged, but not with possession of CSAM.

[robertoandred]

What exactly would you be charged with? Why would law enforcement even be involved in a case of false positives?

[onepunchedman]

The sender would of course be charged with wasting police efforts, defamation attempts+++. In the case of false positives the receiver of course wouldn't be charged, it's more about the fact that this system can be manipulated with too much ease. Even if you're not charged, an investigation takes time away from already limited law enforcement resources. I'm also not interested in buying products from a company that blatantly spies on me. Today it's CSAM, but as others have pointed out, the hashes can be changed to look for anything.

[robertoandred]

Do you mean charging the sender of the trick images or the receiver?

[onepunchedman]

Well that depends on the situation. Regardless the sender would be charged if found, but if they were able to get legitimate CSAM on the receiver's phone the receiver could possibly be charged too, or at least investigated. Just the idea of getting investigated in these kinds of attacks, much less being exposed publicly as being under investigation is a horrible thought.

[robertoandred]

You specifically said someone would be sent known CSAM. How would that get added to their photo library?

[onepunchedman]

I meant * could *. My point is that social engineering is a clear weak link in this system. They can also be sent regular photos whose hash matches the database, or use this repo to transform a regular pornographic photo's hash, making it hard for manual confirmation on Apple's part.

[robertoandred]

What kind of social engineering would lead an innocent person to save known CSAM to their photo library?

[copperx]

None needed. You could just send a photo to the target through WhatsApp, and the photo would be automatically synced with iCloud.

[simondotau]

Wouldn't the photo be scanned for CSAM by WhatsApp first?

[sethgecko]

Whatsapp messages are e2e encrypted so no.

[heavyset_go]

What kind of social engineering would lead an innocent person to install malware on their devices? Or do you think people like that want to take part in an illegal DDoS botnet?

[robertoandred]

I think there’s a difference between “I’ll click this totally legit button to protect my computer from viruses” and “I’ll save this picture of a child being raped to my photo library.”

A lot of people may not know how to avoid malware. But I don’t think very many of them would be so inept as to accidentally long press on child porn and tap “Add to Photos”.

[philipswood]

... and "I'll save this picture of an hilarious kitten to my photo collection"...

Fixed it for you.

The image to be saved doesn't have to be disturbing at all to trigger a hash collision.

The linked repo has code to modify an image to generate a hash collision with another unrelated image.

That's the whole point.

[Dylan16807]

If some commenters can be believed about their experience with the database, there are a bunch of completely innocuous images in it because they're from the same photosets or distributed alongside CSAM.

Is that enough to cause an investigation? Maybe, maybe not, but I wouldn't want it to be a risk.

[simondotau]

Photos in the database are classified for their content. Only images classified as A1 (A: prepubescent minor, 1: sex act) are being included in the hash set on iOS. So this doesn't even include A2 (2: lascivious exhibition), B1 or B2 (B: pubescent minor) let alone images which are in the database and aren't classified as any of A1, A2, B1 or B2.

While I've no doubt that there's a lot of "before and after" images (which are still technically CSAM even if they're not strictly child porn) and possibly many innocuous images, they would not have been flagged as "A1".

I'm sure there's probably still a few images flagged as A1 which shouldn't be in the database at all, but that number is going to be small. How many of these incorrectly flagged images are going to make their way into your photo library? One? Two?

You need 30 in order for your account to be flagged.

[Dylan16807]

If someone is deliberately targeting you with them, 30 isn't very hard to reach.

[onepunchedman]

Lending your phone to someone for a call, then a quick airdrop. Legitimate-looking emails with buttons. There's probably a list somewhere of proven attack vectors.