[andrewmd5]

If you can recreate a file so it’s hash matches known CP then that file is CP my dude. The probability of just two hashes accidentally colliding is approximately: 4.3*10-60

Even if you do a content aware hash where you break the file into chunks and hash each chunk, you still wouldn’t be able to magically recreate the hash of a CP file without also producing part of the CP.

[ornornor]

The Twitter thread this whole HN thread is about shows just how to make collisions on that hash. So any image can be manipulated to trigger a match, even if that image isn’t CP.

[ok123456]

It's NOT a cryptographic hash.

It's the weights from the middle of a neural network that they're calling a "hash" because it encodes and generates an image it has classified as bad. Experts have trouble rationalizing about what weights mean in a neural network. This is going to end badly.

[headmelted]

Exactly this.

If this was a hash then it would be as the parent describes, this is at best a very fuzzy match on an image to take into account blurring/flipping/colour shifting.

It's vastly more likely that innocent people will be implicated for fuzzy matches on innocuous photos of their own children in shorts/swimming clothes than it is to catch abusers.

The other thing is, when you have nothing to hide you won't take efforts to hide it - meaning you'll upload all of your (completely normal) photos to iCloud without thinking about it again.

The monsters making these images know what they're doing is wrong, so they'll likely take efforts to scramble or further encrypt the data before uploading.

tldr; it's far likelier that this dragnet will only even apply to innocent people, than it is to catch predators.

All this said, I'm still in support of Apple taking steps in this direction, but it needs far more protections put in place to prevent false positives than this solution allows. A single false accusation by this system, even if retracted later and rectified, would destroy an entire family's lives (and could well cause suicides).

Look what happened in the Post Office case in the UK as an example of how these things can go wrong - scores of people went to prison for years for crimes they didn't commit because of a simple software bug.

[dragonwriter]

> The monsters making these images know what they're doing is wrong, so they'll likely take efforts to scramble or further encrypt the data before uploading.

The ones that make national news from big busts do, because the ones that don't get caught much sooner and only make local news, because Google and other parties are have automatic CSAM identification online already (server side, not client side, AFAIK), and are sending hits to Homeland Security.

[saagarjha]

Adversarial examples are anything but random.