[modernerd]

The reported response from Apple offers little reassurance:

> The executives acknowledged that a user could be implicated by malicious actors who win control of a device and remotely install known child abuse material. But they said they expected any such attacks to be very rare and that in any case a review would then look for other signs of criminal hacking.

What triggers them to look for signs of criminal hacking?

Does every manual review process involve such checks?

Are they searching device backups for indicators of compromise [IoC]?

What if there's no device backup or device image to scan?

What if the scan fails to notice IoC?

What if the device was compromised after the last backup?

What if the device was compromised via physical access?

What if the device isn't compromised and the material was pushed maliciously or via drive-by download?

It's dangerous to assume that all material on a network-connected device arrived with the consent of the user when it can accept incoming messages from strangers, trick people into downloading files, or be compromised without your knowledge.

“That isn't mine” is going to be a tough defence if you can't even take measures to log where content came from.

Client-side scanning seems to amplify this issue (which could still happen with cloud storage) because at least cloud storage doesn't generally ship with or integrate deeply with messaging apps, social media, a web browser, QR codes, App Clip Codes[1] etc.

The impact might be fairly low right now with the current proposal (images would have to be uploaded to iCloud, so cached browser images don't get scanned as far as we know), but the existence of the non-consensual scan in the first place is worrying, because it means such attacks are only a policy change away.

[1] : https://developer.apple.com/design/human-interface-guideline...

[GeekyBear]

> The executives acknowledged that a user could be implicated by malicious actors who win control of a device and remotely install known child abuse material.

Since Google has been scanning your account for kiddie porn for the past decade, wouldn't this apply equally to Google accounts?

>a man [was] arrested on child pornography charges, after Google tipped off authorities about illegal images found in the Houston suspect's Gmail account

https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-le...

All people have to do is email you kiddie porn and Google will have you arrested?

[modernerd]

No, but the person who sent that message could get in trouble.

In the case you linked to the person was reported for sending email to a friend with attached CSAM, not for receiving it.[1]

Apple's system scans images client-side if they're due to be uploaded to iCloud. That process can happen without user consent or action. For example, WhatsApp and other messaging apps save images to photos, which are auto-synced to iCloud. (If you use WhatsApp and iCloud you'll find your Photos section full of memes from WhatsApp group chats when you log in at icloud.com, for example. This was a surprise to me at first.)

So the risk of malice seems higher with Apple's system than with the long-running PhotoDNA implementations backing Gmail/Google Drive/OneDrive etc.

Gaining access to someone's email and sending attached CSAM is likely to cause them more issues than receiving it. But that's harder because you need their login info and not just their email address/phone number, which is all that an attacker potentially requires to trigger action from Apple's automated scans.

[1]: https://nakedsecurity.sophos.com/2014/07/31/google-tips-off-...

> The investigation was apparently sparked by a tip-off sent by Google to the National Center for Missing and Exploited Children, after explicit images of a child were detected in an email he was sending.

[GeekyBear]

> No, but the person who sent that message could get in trouble.

Is there some reason to imagine the person sending the message couldn't do so with burner email accounts or by abusing open/vulnerable email servers?

Has Google suddenly prevented spam from landing in your spam folder without anyone noticing?

It's much simpler to send email than it is to take control of someone's device.

[modernerd]

Right, the sender isn’t going to use their own email address in an attempt to incriminate you. My point was that receiving material by email from a stranger doesn’t make you liable for its contents (unless there is a record of you requesting the content). It makes the sender liable (if they can be traced).

Apple’s approach does not seem to provide the same safeguard. Your account will be flagged for review if there are n flagged images destined for upload on your device. The description of the process does not mention if or how provenance or intent to receive those images is established.

[Clubber]

I mean you think that would be how it works, but say a system found the image stored in your mail's temp directory and notified the police, do you think they would be that interested in finding the person who sent it, or do you think they would think, "You had kiddie porn on your phone, that's against the law. 30 years." Win.

[ant6n]

Is there even a way to get iCloud or Google photos on the iPhone to only upload photos taken with the camera, to not spam one's photo account with chat garbage?

I was trying to figure out a way, but got side tracked on the issue, then my phone got stolen and I lost a bunch of family/baby pictures (thanks Google/apple).

[modernerd]

WhatsApp has a setting you can disable:

Settings → Chats → Save to Camera Roll

Not sure about other messaging apps.

[jxdxbx]

Google Photos are also scanned. Just after they're uploaded.

[zionic]

How do we know they aren't?

Anyone with your credentials to social media/any cloud service like gmail could send CP on your behalf to get you flagged and interrogated.

Good luck mounting a defense against a subject this taboo. Even if you win it will follow you forever.

[diebeforei485]

Gmail blocks incoming messages that contain CSAM, so you don't actually have this concern. It's similar to if someone tries to send you an email with an attachment that has a computer virus. It will never reach your Gmail account - not even your Spam folder.

(In the virus case, they also do a second scan when you open the message - with updated virus definitions to catch new viruses).

[ipaddr]

I thought google wasn't scanning your emails anymore.

[GeekyBear]

Wasn't that claim limited to children's accounts?

Since Google is saving a history of what you purchase from third party merchants by scraping invoices and receipts sent to you through your Gmail account, it's safe to say that they are scanning your emails.

https://news.ycombinator.com/item?id=26248486

[jxdxbx]

Google scans all photos in the cloud (Gmail, Drive, Google Photos) for CSAM and has for a long time. It just doesn't show contextual ads against email anymore, since those all sucked.

[fortran77]

> “That isn't mine” is going to be a tough defence if you can't even take measures to log where content came from.

It's not a defense at all. This material is prosecuted under a "strict liability." It doesn't matter how you got it, you're liable.

[35fbe7d3d5b9]

> This material is prosecuted under a "strict liability." It doesn't matter how you got it, you're liable.

You're overselling it.

First, there is a statutory affirmative defense: if I obtain CSAM and "promptly and in good faith" delete it or report what happened to law enforcement, liability does not attach.

Additionally, federal laws are clear that you have to knowingly receive CSAM. That's not just a legal flourish or a word – knowledge is an element that a jury or judge will rule on. If I ask you to send me an illegal video and you do, we've both knowingly violated federal law. If you send me to a webpage that purports to offer me a job, but actually has images hidden with CSS to poison my cache, I've not knowingly received anything.

[vineyardmike]

> knowledge is an element that a jury or judge will rule on.

And yet, i never want to be in this court case at all.

[artificialLimbs]

>> they expected any such attacks to be very rare

Very rarely will your life be completely ruined based on inaccurate information.