[simondotau]

None of those attacks would work against the system as described by Apple. The only photos scanned are items in your photo library prior to upload to iCloud. Your browser cache is not scanned.

Hash collisions would fail human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM.

[T3OU-736]

> Hash collisions would not pass the human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM.

I truly wish I could subscribe to this optimistic view. Experience tends to show this to be unlikely.

Two factors combine against it: 1. There is no negative consequence for a mis-flag (to the reviewer) 2. This set up is a tool, and like many tools, inventive humans will find a way to subvert it in the name of convenience. I am referring to NSLs from U.S. Patriot Act as an example. Since CSAM is such a toxic thing (let's stipulate that CSAM itself is unequivocally bad), there is less tendency to examine it closely for, well, CSAM-ness.

[simondotau]

Again, I'm only pointing out how this conflicts with Apple's description of their system. I'm in no position to know whether their description is accurate or how it will actually operate in the real world.

For the sake of argument, let's assume you're correct and Apple's review team are lazy shits who don't look at the images. Okay, so Apple then sends the report onto NCMEC. What are they going to do when they open the report and it turned out the images Apple reported were hash collisions?

[lvh]

My understanding (from someone who would know but said this in a Chatham House rules space) is that NCMEC is already incredibly underfunded, understaffed, and backlogged. Similar incentives apply to them. They're a nonprofit: a private organization who has significantly fewer dollars than Apple does.

[simondotau]

The critical follow-up question is what do NCMEC do with their backlog? Unless they're dumping this backlog directly at the feet of law enforcement, I don't see how this changes the equation.

[laurent92]

All watchers of Clara Morgan were watching what is legally categorized as “child porn” (=“any depiction of an individual under 18).

And since “depiction” includes drawing, any consumer of Hentai (s. manga) is hosting what passes legally as clear child porn.

I wouldn’t be surprised if 25% of the youth could be taken to jail according to the law, so, definitely, a learning period or warnings are required.

It’s akin to all the US adults who are registered as sex offenders because they peed in a park at night. Apple is clearly help with law abuse here.

[simondotau]

That may be true in principle, but irrelevant with respect to Apple's CSAM process. Unless the exact material is explicitly catalogued by NCMEC or another child safety organisation, there won't be a hash match.

This isn't a porn detector strapped to a child detector.

[caseyscottmckay]

Do you have a source for a single person being required to register as a sex offender for peeing in public?

[shakna]

Peeing in public is often charged as indecent exposure, which can have you forced to register as a sex offender. [0][1]

It doesn't take long to find those cases.

[0] https://www.nevadaappeal.com/news/2021/mar/21/public-urinati...

[1] https://law.justia.com/cases/california/supreme-court/3d/10/...

[saiya-jin]

You are rather desperately trying here to downplay a massive security fuckup by Apple as if its perfectly fine. One of the main selling points of Apple, heck for many the most important one, was just blown to pieces couple of days ago. Its NOT Okay for Apple to send your images further.

The only argument left missing here is 'you have nothing to hide anyway, right?'.

I would be able to accept an inferior OS incapable of true multitasking and with very limited options to set. Closed system with no sideloading. I would even accept a lousy zoom on flagships cameras compared to, well, any competition. Proprietary connection port. Mediocre battery life. Overpriced accessories. But start removing security, and that's one step too far.

[simondotau]

I was assuming for the sake of argument. I am not saying that a "major fuckup" of Apple's human review process would be acceptable.

[veidr]

    > Hash collisions would fail human review.

This (pervasive, over the past couple days) idea that Apple (of all major tech companies, lol!) will be capable of manually reviewing tens of thousands of automated detections per day is... nuts.

The "system as described by Apple" doesn't comport to reality, because it relies on human review. If you remove the human review, the system is fucked.

But no company on the planet has the capability to sanely and ethically (to say nothing of competently or effectively) conduct such review, at the scale of iOS.

[TeMPOraL]

Can they even, legally, review anything at all? I mean, it's highly likely there will be actual CP among the matches, viewing of which is - AFAIK - a crime in the US.

[cmsj]

That is somewhat unclear at the moment. They don't get to see the actual image in your library, they see a derived image that's part of the encrypted data uploaded by your phone as it analyses the images.

I don't believe any of the information they've released thus far, gives any actual detail about what that derived image actually is.

One might guess it's a significantly detail-reduced version of the original image, that they would compare against the detail-reduced image that is able to be generated from the matching hash in the CSAM database.

[simondotau]

Tens of thousands of automated detections per day? Unlikely. More likely tens per year. Remember, this isn't a porn detector combined with a child detector. It is hashing images in your cloud-enabled photo library and comparing those to hashes of images already known to child abuse authorities.

In addition, consider how monumentally unlikely it is for any CSAM enthusiast to copy these illicit photos into their phone's general camera roll alongside pictures of their family and dog. This is only going to catch the stupidest and sloppiest CSAM enthusiast.

[detaro]

For comparison to your "likely tens per year" number, Facebook is running the same kind of detectors and reports ~20 million instances a year: https://twitter.com/durumcrustulum/status/142377627884745113...

[scbrg]

That doesn't seem to be the same kind of detectors at all.

"21.4 million of these reports were from Electronic Service Providers that report instances of apparent child sexual abuse material that they become aware of on their systems."

So those 20M seems to be images that Facebook looked at and determined to be CP. Apple's system is about comparing hashes against already known CP.

For the record: I don't support Apple's system here, but it's not the same kind of detection at all. Let's try to not make up random facts.

[detaro]

From the same thread: https://twitter.com/alexstamos/status/1424017125736280074

> The vast majority of Facebook NCMEC reports are hits for known CSAM using a couple of different perceptual fingerprints using both NCMEC's and FB's own hash banks.

[scbrg]

Ah, I see. My apologies.

[DannyBee]

Facebook looked at them after they hash matched known CP. That is how all these providers do it.

If you think that this is 20 million people mashing the report button, that is almost certainly wrong

[simondotau]

That's a summary number of many kinds of reports, of which CSAM hash matches would be one part.

That summary number also includes accusations of child sex trafficking and online enticement. I wouldn't be surprised if reported allegations of trafficking and enticement were in excess of 99.9% of Facebook's reporting. But since they don't break it out, I can only guess.

Given that guesses aren't useful to anyone, it would be interesting if you know of any statistics from any of the major tech vendors, of the reporting frequency of just CSAM hash matches.

[detaro]

> of which CSAM hash matches would be one part.

The majority part:

https://twitter.com/alexstamos/status/1424017125736280074

> The vast majority of Facebook NCMEC reports are hits for known CSAM using a couple of different perceptual fingerprints using both NCMEC's and FB's own hash banks.

[simondotau]

Fascinating. Thank you for providing the clarification. I still find that number to be perplexingly huge. If it's indeed correct, one hopes that Apple know what they're getting themselves in for.

[matwood]

Google is probably a better comparison. I can't find the source atm, but IIRC it was ~500k/year.

[simondotau]

That wouldn't surprise me as Google's reporting would include everything seen by GoogleBot as it crawls the internet.

[cgio]

Ten thousand iOS users doing something stupid or sloppy per day (noting they don’t have to be stupid or sloppy in general for that to happen) would not hit the monumentally unlikely criteria for me. Also this is not counting the false positives which is the premise of this thread.

[simondotau]

Yes, being sloppy is common.

I don't know about anyone else but I've never had any issue with regular porn sloppily falling into my camera roll. And that's just regular legal porn. Maybe I'm more diligent than others but regardless, it's just not something that happens to me.

Being sloppy with material which you know is illegal? Material which, if stumbled upon by a loved one, could utterly ruin your life whether or not authorities are notified? Material which (I optimistically assume) is difficult to acquire and you'd know to guard with the most extreme trepidation? We're seriously expecting tens of thousands of CSAM enthusiasts to be sloppy with their deepest personal secret and have this stuff casually fall into their camera roll?

I'm not buying that.

[samrolken]

A false positive will not have any effect. The threshold system they have means that they won’t be able to decrypt the results unless there are many separate matches.

[soziawa]

> Hash collisions would not pass the human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM.

The whitepapers provided by Apple do not say what the human reviews consists of. They could just look at the hashes to make sure there isn‘t a bug in their system.

[simondotau]

> The whitepapers provided by Apple do not say what the human reviews consists of.

At minimum what we know is that each flagged image generates a "safety voucher" which consists of metadata, plus a low-resolution greyscale version of the image. The human review process involves viewing the metadata and thumbnail content enclosed in each safety voucher which cumulatively caused that account to be flagged.

[foobar33333]

A human at Apple likely doesn't get access to anything. I assume it would be part of the police group under strict restrictions checking these.

[simondotau]

The data is not sent to a "police group", it is sent to NCMEC.

From Apple's FAQ:

Will CSAM detection in iCloud Photos falsely flag innocent people to law enforcement?

No. The system is designed to be very accurate, and the likelihood that the system would incorrectly flag any given account is less than one in one trillion per year. In addition, any time an account is flagged by the system, Apple conducts human review before making a report to NCMEC. As a result, system errors or attacks will not result in innocent people being reported to NCMEC.

[douglasisshiny]

NCMEC then makes those images available to the appropriate law enforcement agency after the fact.

[simondotau]

Yes, if they're CSAM.

[zionic]

This is out of date. It took less than a week for Apple to announce this tech is coming to "3rd party apps".

[GistNoesis]

One obvious problem with human review is steganography.

The picture can look normal to the human eye, but if it contains hidden content (in the least significant bit of each pixel for example so that the hash is unchanged), a forensic software will definitely notice, raise some flags, and extract the hidden offensive content automatically, leaving the human reviewer no other choice but to report you.

If Apple says they are not going to look for hidden content, then they are just handling a free pass which render the whole scanning thing pointless.

[simondotau]

I'm confused what scenario you're positing here. Given the widespread adoption of encrypted communications, steganography is of no use to traffickers of CSAM. Steganography generally serves only one purpose, which is to transfer material in public view with plausible deniabilty—such as leaking material out of a military facility which has exceedingly robust data protection processes.

Apple have explicitly said that their hash algorithm is only concerned with visible elements of the image.

[GistNoesis]

I'm speaking about the adversarial scenario of an attacker trying to frame a target. He just need to get on your phone an image with hidden content that has a hash collision with the database.

Traffickers and consumers of CSAM know that their content is illegal to possess and store so they sometime use steganography software to store the offensive data inside their innocuous photo library. This way when they can browse their private collection via the lens of the steganography software and they don't have some suspicious encrypted file that would attract attention of someone they share the computer with.

[simondotau]

You seem to be confused. As you said yourself, steganographic concealment would, by its very nature, not change the perceptual hash of the visible image. If the visible image doesn't match an known hash, the steganographically modified version isn't going to either.

[GistNoesis]

This sit on top of the perceptual hash collision.

First you generate an innocuous image that has a bad hash collision. (This is easy because perceptual hash are not cryptographically secure). Then in a second step you hide some offensive content in it via steganography without changing the hash. Then you send the image to the target.

He stores it in his cloud, it gets flagged because of the hash collision, so it get a manual review. The manual review take the image through some forensic software, which will catch the steganography (because the attacker will have chosen a weak scheme) which will reveal the hidden offensive content and then report you.

[simondotau]

The manual review process only involves a severely transformed (low resolution, greyscale) version of the image which is attached to the safety token. The ability to decrypt any original files only occurs if the human review process confirms the presence of CSAM.

[madmoose]

What you're talking about here has nothing to do with what Apple is implementing.

[jann]

Messaging apps like WhatsApp will save to your photo library though (unless disabled).

So any photo sent to you would be scanned. If you someone sent you a bunch of files, that might trigger a manual review, that would most likely flag your account.

I wouldn't expect that immediately deleting them would stop the review process.

[jbverschoor]

again, I hope someone sends a couple of executives the recently posted images, to make a point

[themaninthedark]

That is why they talk about having a manual review process. So that when someone wealthy or politically connected triggers the system there is a review.

[simondotau]

I haven't used WhatsApp, but I'm tempted to call bullshit on that. I've never used any messaging app on iOS which saves photos to your photo library. Doing so would make no sense and would surely be infuriating. It's also worth noting that apps on iOS can't save to your photo library unless you give them explicit permission.

[tpush]

WhatsApp does by default save received images to your photo library (as opposed to e.g. iMessage). You can turn that off, though. And the permission to read from a user's photo library (to e.g. post images) includes the ability to write to it.

[simondotau]

Gross. I can't fathom how anyone would put up with that.

[kemayo]

WhatsApp really does it, by default. It's a weird choice.

https://faq.whatsapp.com/iphone/how-to-save-incoming-media/

[themaninthedark]

It depends on how the attack was crafted:

Step 1: Get copies of pictures of targets kid in bath from phone/SNS

Step 2: Manipulate pictures so that hash collides with CSAM

Step 3: Get pictures back on targets phone so they get scanned.

If it were me, I would try and get a series of photos from the target, and manipulate several that look most borderline. That way it looks like more than a one off.

Now if there is an Apple review, the person who views them will see some suspect pictures and would confirm.

Now the target would have to get someone to review the original pictures vs the modified pictures. Good luck with the defense.

[alpaca128]

> Hash collisions would fail human review

You mean like the absolutely perfect human review of appstore content that's known for both false positives and false negatives?

Neither automatic nor manual (human) review works 100% reliably. And believing otherwise will only ruin lives.

[cmsj]

You are absolutely correct that neither automatic nor manual review is ever going to be 100% accurate.

I would like to believe though that for this system to fully fail an innocent person, the following would all need to have failed:

1) Coincidental CSAM hash collision 2) Incorrect manual review by Apple 3) Incorrect subsequent review by NCMEC 4) Inability of a lawyer to obtain the original image for presentation during a trial/appeal

which seems kind of unlikely? (although it's certainly the case that once steps 1, 2 and 3 have failed, the person's reputation is likely damaged even if they are able to prove their innocence in court).

The wider question here is, should 100% accuracy be the bar by which we judge this? I don't think we expect the law enforcement system to be 100% right, hence principles like the presumption of innocence and right to appeal, and even then it gets things wrong sometimes.

[alpaca128]

There are known cases of police faking AI-generated evidence[0]. There's no reason why Apple would be immune against such things. And the recent British post office scandal shows that even without manipulation false faith in technology as evidence can destroy hundreds of lives. The low chance of an error going through that whole chain of checks also increases the trust in that system even in the case of a false positive.

And all this is assuming it will never be expanded from CSAM to other content. Apple is already rolling out a censored version of iOS in China.

[0] https://www.vice.com/en/article/qj8xbq/police-are-telling-sh...

[samrolken]

You’re missing the threshold that is part of this system. You would need multiple hash collisions across multiple photos to trigger these mechanisms.

[zionic]

Of course not, Apple will simply match the much more reliable Youtube flagging system :P