[laurent92]

Backup is a good point:

- Apple: “Backup your phone to iCloud, it will be safe there.”

- 5 minutes later: “We’ve wiped your account because of a photos of (porn actor here) which is not CP but technically minor at the time she filmed.”

- “Also we’ve wiped your iPhone because we couldn’t knowingly let you keep that. Good luck contacting your parents, we’ve deleted your contacts. Good luck! PS: We’ve reported you to the police.”

- Also you can’t connect to your iMac now.

[annamargot]

Or photos of your own children.

We have a Tumblr set up for family to view pics of the kids. Several photos and videos of our kids when they were under 2 were taken down either temporarily or permanently by their CP algo.

These were a pic or video of kids in the bath or without a shirt. In none of them could you see bum or bits. Just a semi naked baby.

Algorithms like this get things wrong all the time

[tpush]

This is not the kind of algorithm that Apple is be using. That one only scans for already known CSAM in NCMEC's database.

[ElFitz]

Quite funnily and disturbingly, one the databases of "known CSAM" hashes also apparently includes a picture of a clothed man holding a monkey[1]

[1]: https://www.hackerfactor.com/blog/index.php?/archives/929-On...

[severak_cz]

That was just a MD5 collision - an image that has same MD5 hash as some other image (in this case some CP). This is uncommon yet possible thing - see this example[0].

[0] https://natmchugh.blogspot.com/2014/11/three-way-md5-collisi...

[tyingq]

I think a flawed process where the monkey image ended up in the database is more likely than a random unintentional hash collision.

[zarzavat]

Not really. MD5 is thoroughly and completely broken, and has been for years. You can modify an image to be an MD5 collision for another image.

[ElFitz]

Yes, hash collisions definitely occur. There is no such thing as collision-free hashes, and MD5 is definitely broken.

Even though the author says they were 3 million MD5 hashes the second time, the first one he calls them SHA1 and MD5 hashes (even though SHA1 is considered weak too).

I wonder what kind of hashes Apple is planning to use. Will it be whatever is made available to them or will they only accept (what is now considered) secure standards?

[shakna]

Which may contain the hashes of their photos, because they've been taken down in the past, which means they probably have been added to certain blacklists that may have been integrated into the blackbox of NCMEC's database.

[simondotau]

Photographs of your naked child in the bath are not illegal, are not CSAM, and are not going to be in the NCMEC's database.

[shakna]

NCMEC's CSAM database already includes images that are not necessarily illegal. If _your particular_ photos have been flagged in the past, they may well be part of the database.

[culturestate]

> NCMEC's CSAM database already includes images that are not necessarily illegal.

How could this be the case? If it's been determined to be CSAM then it is, by definition, illegal.

If it were true that the database is likely to contain legal material, how would we possibly know about it, given that the contents of the database are secret?

[simondotau]

If you're sending other people photos of your children that are explicit enough to prompt someone bring them to the attention of child safety groups like NCMEC, and they look at it and agree it's worth their time to investigate, the first you hear of it isn't likely to be after it eventually comes full circle through Apple's CSAM processes.

Remember, this isn't a porn detector strapped to a child detector.

[themaninthedark]

Step 1: Get copies of pictures of targets kid in bath from phone/SNS

Step 2: Manipulate pictures so that hash collides with CSAM

Step 3: Get pictures back on targets phone so they get scanned.

I don't have the skills or understanding of how the hashes are created but would this be possible?

[simondotau]

Hypothetically that's possible, although all three steps you listed are exceedingly non-trivial. The notion that an attacker could pull off two of those steps let alone three is borderline fanciful. In addition, their target must also qualifies with the necessary prerequisites:

• has an iPhone;

• has children;

• took photos of their children which could be mistaken for CSAM by a sloppy reviewer;

• is of sufficiently high importance to justify the effort.

And after that insane effort, all you've done is inconvenience your target for a little while until child safety people investigate your family situation and discover that the photos which got flagged were not actually CSAM.

Immediately after the investigation process discovers the hash fraud, Apple will immediately start delving into exactly how their hash algorithm failed in this instance, improving it to mitigate this exploit. So this target better be worth it!

If this was a plausible exploit, surely it would have already happened to people with Android phones since Google has been doing pretty much the exact same scanning of customer images for over five years. (The only difference with what Apple is now doing is where the hashing is performed—but this makes no functional difference to the viability of your hypothetical exploit.)

[simondotau]

This isn't an ML algorithm. It's a hash. It only matches already known material.

[samrolken]

It is a hash created with ML. So it’s both. But yes, it only matches already known material.

[simondotau]

I haven't seen anyone claim that any of this algorithm was "created with ML". I'm interested in learning more so do you have a citation for that?

Regardless, it's not both. Setting aside how the algorithm was created, it's incorrect to say that an algorithm "created with ML" is itself an ML algorithm.

NeuralHash was so named because it was optimised to run on the Apple Neural Engine for the sake of speed and power efficiency.

[samrolken]

It’s both because it’s a multi-step process.

The image is not fed directly into the hashing function, like taking an MD5 hash of a file or something.

Rather, the image is first evaluated by a neural net that looks at specific visual details, and has been trained to match even if the image has been cropped or anything like that. The results of the neural net evaluation are what is then input for the hashing function.

This is explained in detail in Apple’s documentation they released with the announcement.