[ianmiers]

This is a harmful distraction from the massive issues with Apple's proposal. If you wanted to frame someone for possession of CSAM, similar stunts can be pulled with Google, Facebook, Instagram, and Microsoft today. Yes, the scope here is broader and some people don't use any of those, but ....It's silly, and it makes the tech community look like a fringe minority of screeching conspiracy theorists.

And this is a problem because Apple's proposal is really really awful. Apple is normalizing scanning your private phone for files and reporting them. They built the technical capability to do it for any photo, and they will be under enormous pressure to expand it both in the US and abroad. And the fact that they did it will be used to pressure other companies into doing the same and to legitimize laws that require scanning for any content the government can justify.

Apple built a surveillance mechanism that is incredibly powerful. One no government could ever force a company to design and build. But once it's built, the only thing stoping it from being abused is Apple's pinky promise they won't let it happen. If you believe that legal norms, big tech companies and some quasi governmental nonprofit like NCMEC will stop such an abuse if it happens .... where have you been living the past few years? Because it sure isn't the US, the UK, Turkey, or China.

[skybrian]

Is this actually different from other cloud photo apps? If you use Google Photos then your photos will likely be scanned by Google. If you use Apple’s photo app then their app will do the scanning.

There seems to be a vague idea floating around that this is built into the OS or the device just because the scanning happens on the device, but it’s not clear that’s the case. Apple doesn’t make the distinction between OS and app clear either.

[ianmiers]

Factually, not yet. That will change and I will explain how in a moment. But first there's a major difference between doing it on device vs in the cloud. It changes how we think about privacy and builds a capability to scan phones (not particularly limited to iCloud) into the device. That's a capability no Western tech company could ever be forced to build for more illicit usages, but now it exists.

Second, Apple almost assuredly will encrypt iCloud after this. So now we have the precedent of scanning encrypted messages. And that will then feed legislation that congress has been attempting to pass for years to kill any right to meaningful end to end encryption for messaging. https://blog.cryptographyengineering.com/2020/03/06/earn-it-...

[shadowgovt]

Ironically, the difference is that Apple is doing it at the client layer so that they can't do it at the server layer; the user's iCloud [edit: photos, not all of iCloud] is encrypted at rest against Apple accessing it.

This approach makes mass-sweeping of all server-side stored data harder to accomplish (whereas in, say, Google Photos, Google can break-glass server side to get into someone's private data, so they could hypothetically do a mass-scan if the government demanded it).

[nightski]

Right but it's easier to just not use Google Photos. It's harder to opt out of your phone. I realize they "said" that device scanning will only be used if iCloud is enabled (right?). But ToS changes constantly and who knows what the future holds.

[nicce]

It is only applied for photos which are going to iCloud. If they change that, then we should be really worried. Current method is only pure improvement if leave all speculation out of it.

[ribosometronome]

I am generally in agreement with you (and have made similar arguments, if you look at my post history), but the "expand in unspecified ways" is a bit ominous. Committing to only scanning photos that are being synced to the cloud (effectively, keeping parity with what everyone does, just doing it on device at the time of upload instead of in the cloud) would be really welcome here.

[hypothesis]

Will such commitment substantially change anything?

First line of their privacy policy is: “Apple is committed to your privacy.”

[1] https://www.apple.com/legal/privacy/

[skybrian]

It's not at all hard to avoid using an app on your phone. I have an iPad and I use Google Photos. I've never used Apple's photo app.

This is what I meant by mixing up (and blurring the lines between) app-level and OS-level capabilities. It might not actually be mixed up technically, but it seems to be the user perception.

[ianmiers]

Yes, and even more ironically, thats precisely the problem. Because it makes mass sweeping of client side content viable --- both technically and morally--- in a way never possible before. The only thing stoping scanning of the entire phone for anything, now that Apple built the technical capability, is Apple's willingness and ability to resit pressure from the US, UK, China, and others to use it.

[shadowgovt]

Very true. Of course, that's always been true since they manufacture the hardware and the OS for the hardware. They're optimally positioned to hide any type of behavior they want in the full stack of the product.

The only thing stopping your phone from keylogging your password to a server in the NSA somewhere if it recognizes a specific trigger pattern is Apple's willingness and ability to resist pressure from the US, etc.

[ianmiers]

>The only thing stopping your phone from keylogging your password to a server in the NSA somewhere if it recognizes a specific trigger pattern is Apple's willingness and ability to resist pressure from the US, etc.

Think of what would happen if you tried to make your average Silicon Valley dev team design, implement, and test a surveillance system they didn't want to build and that was immoral. They'd resit in an infinite number of ways that would delay the project virtually for ever. Short of summary executions, I bet you could not get a nice, efficient, effective system.

On the other hand, once the dev team has enthusiastically built the system that scans for any image, it's entirely easy to say "Now, make it look for these images." They have no avenue for resistance other and a up front no. And a government that wants to do totalitarian things knows many ways to force a yes.

[shadowgovt]

Apple (and the other FAANGs) do not employ average Silicon Valley dev teams.

In general, a company at that size would approach this problem by figuring out who in the company is willing to take on an unsavory challenge like this and then forming a skunkworks out of them, slightly sequestered from the rest of the company.

I'm not saying Apple has done it, or that they're incentivized to. But it's trust-turtles all the way down. Either we trust them to say "No, you can't use our tech to harm our users," or we don't.

[McFlummoxed]

> Think of what would happen if you tried to make your average Silicon Valley dev team design, implement, and test a surveillance system they didn't want to build and that was immoral.

It wouldn't be that. It would be defense contractors sitting at Lockheed or a few blocks from DARPA whose daily bread is making a Tech Sandwich whenever the Broad Agency Announcement for one shows up on sam.gov, or on the DARPA page, or the variety of procurement sites that the government doesn't expose to the internet. If they want it, they can get it -- no persuasion of liberal tech-bros needed.

[Klonoar]

iCloud as it currently stands is not encrypted to where Apple can’t access it.

[shadowgovt]

Good catch; I should have said just the photos. Backups and some other pieces are not end-to-end encrypted and stored encrypted at rest.

Updated original comment.

[nicce]

There is actually evidence (iOS 15 beta), that they added option to recover your backup from recovery keys. This strongly suggests that E2EE is coming.

Someone was worried about how they handle the keys. They have solution for that already: https://blog.cryptographyengineering.com/2016/08/13/is-apple...

[oarsinsync]

> the user's iCloud [edit: photos, not all of iCloud] is encrypted at rest against Apple accessing it.

This is false. They present a web interface showing the photos. The UI isn’t locally generated entirely using JavaScript to decrypt the data. They only way this can happen is if Apple has the decryption keys.

iCloud Photo Library has never been private. Apple has always been able to view your photos.

[shadowgovt]

You are correct; I was misinformed.

https://9to5mac.com/2021/08/05/report-apple-photos-casm-cont...

Apple has the keys; the data is encrypted at rest and in transit, but they can be compelled to use them.

[therein]

I worked at Apple on iCloud and yes, photos were never encrypted. Or should I say blade runner. :))

[spoonjim]

How can the photos be encrypted at rest where Apple can't access them? If I buy a new iPhone all of my iCloud photos show up on it. That means that Apple can access them somehow.

[ummonk]

While photos aren’t end to end encrypted (at least today), the fact that they show up on a new phone isn’t proof that if non-encryption. E.g. keychain passwords and iMessage messages are end to end encrypted (except in iCloud backups) but show up when you buy a new phone.

[ummonk]

The danger is that this is further down the slippery slope.

[nicce]

> Apple is normalizing scanning your private phone for files and reporting them.

”Antivirus cries in corner as forgotten...”

I know, iOS has no build-in AV (like MacOS) but still, it is a bit laughable that many existing tools provides this same power, and only now it is a concern. On a black box system. I am resilent, and I will join into mass of pitchforks and torches only when there is actual evidence of them expanding their promises or using these features for something else they are meant. They knew the risks when bringing this feature and know the cost when it is proved to be misused.

[ianmiers]

yes, you can turn AV into the same thing. But no one has been advocating for that or passing laws that would make AV both mandatory and have to report you to the police.

This has been going on for CSAM scanning for a while now. The latest version was called the EARN IT act [0]

The argument is not this is a slippery slope where you might miss step. IF that was the case, yes AV would be analogous. Instead, its that people are actively trying to push you into the spikes at the bottom of the pit, don't build things at the edge of the pit where the handrail is a pinky promise not to let others push you.

[0] https://blog.cryptographyengineering.com/2020/03/06/earn-it-...

[nicce]

> don't build things at the edge of the pit where the handrail is a pinky promise not to let others push you.

People are afraid, that the use of these tools are expanded in secret (hidden) for more than they should. From that point of view, legislation motives and discussion does not matter, because capability is valid for many tools at any moment, hence same speculation has applied before.

However, if you want to apply surveillance publicly, then we indeed need legal base for pushing specified tools as mandatory. To expand it for more than CSAM, it will be quite slow process, and implementing something like that publicly before legal base is path for descruction for any company, because people can change for other company.

Is Apple now making that legal process faster?

While it feels like Apple is now closer to the edge of the pit, from techical perpective there is no difference yet. Tools have existed and system is closed source. Question is still the same; ”would you spy for us”? I don’t think that answer has changed from Apple because they changed the location of the image scan.

So, the question is, will legislation change towards more surveillance. Whatever the result is, I think it would have happened whether Apple added this feature or not, as it is not morale excuse. People find the way. In China, it is simply illegal to not install some app by muslims.