[avidiax]

> It is crazy people think apple isnt on the side of privacy.

> It also ensured pressure from governments and plaintiffs, including CSAM victims, who preferred Apple’s more interventionist approaches, which Apple had voluntarily demonstrated it was willing to do.

I feel that Apple open pandora's box with the client-side scanning. It proved that it was technically feasible, and was "privacy preserving". I use scare quotes there because I don't think that political or religious dissidents would find that the same or similar technology used to discover and persecute them is "privacy preserving". And that's really the problem with Apple here. They provided a model for scanning for any kind of message or material while purportedly maintaining privacy.

[bayindirh]

> It proved that it was technically feasible, and was "privacy preserving".

Didn't their paper disproved by reversing the perceptual hashes to reveal blurred version of the images being hashed, and Apple basically said "that's fair, it's not as robust as we wanted, let's visit this later"?

If not, I'll happily stand corrected, but please share sources.

Addenda:

- Apple's original paper: https://web.archive.org/web/20210807165030/https://www.apple...

- Paper breaking the hash: https://arxiv.org/abs/2111.06628

Edit: The second one is the wrong paper. I’ll find and link the correct one tomorrow. Keeping the link for transparency.

[comex]

The paper you linked doesn’t reveal blurred versions of the images being hashed. It does train a classifier to determine which of 1,000 ImageNet classes an image belongs to, which “achieved a top-1 test accuracy of 4.34%”.

[bayindirh]

Then, that’s the wrong paper. I’ll find it and link it as a reply to this comment. Probably tomorrow morning.

[yogorenapan]

> to reveal blurred version of the images being hashed

Skimmed your linked paper. It seems they were able to classify hashes up to ~8% top-1 accuracy and ~30% top-10. Not exactly a blurred version, or any images at all.

So for example, they can say that you probably have images of trees, or images of buildings, but without much other data & very low accuracy.

I'd still be a lot more concerned about them simply flagging political images rather than trying to get a broad understanding of what type of photos I have

[soulofmischief]

It starts at a broad understanding and ends with people permanently giving up their right and ability to keep rogue corporate governments in check.

[duskwuff]

There's actually been some work to reverse the original PhotoDNA hash, with much more precise results than for Apple's NeuralHash:

https://www.hackerfactor.com/blog/index.php?archives/931-Pho...

https://anishathalye.com/inverting-photodna/

[esnard]

Could this be the correct link?

https://arxiv.org/abs/2412.06056

[GeekyBear]

> I feel that Apple open pandora's box with the client-side scanning.

That box has been open for years now.

Big brother is already watching what you do on your Android device.

> A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal.

https://www.nytimes.com/2022/08/21/technology/google-surveil...

[letmevoteplease]

This is not client side. Images just sitting on your Android phone are probably safe (although Google could push an update at any time). Your images get scanned when you back them up, send them over RCS, etc. I have even seen criminal cases originating from reverse image search - anything that touche the servers of the big tech companies, except apparently Apple, will be scanned using questionable AI and against a secret list to Protect the Children.

[GeekyBear]

This is an image that he did not send off of his device to anyone except his doctor's office, yet Google reached into his private data and scanned it anyway.

Google reported him to the police based on a single false positive.

To add insult to injury, even after the police contacted Google to tell them that they had cleared him of wrongdoing, Google refused to restore access to his account.

[izacus]

Stop making stuff up man, the image was uploaded to Google Photos servers.

> The father uploaded photos of his son’s genitals, which were also backed up on his Google cloud, to the health care provider’s messaging system as requested.

[GeekyBear]

Did he set up his device to upload his private data to Google, or did Google create an OS that automatically sent everyone's private data to their AI server for scanning without explicit consent?

[izacus]

Google Photos does ask you for consent when you run it.

(It is, granted, a bit pushy about it and will ask multiple times when you run it with an intrusive dialog.)

[inigyou]

You shouldn't trust any Android phone not running GrapheneOS or similar.

[abrugsch]

VW doesn't trust graphene to allow their apps to run on it :(

[xboxnolifes]

I'm pretty sure this article if from after Apple introduced (floated the idea of?) client-side scanning. I'm not sure it really bolsters the idea that it's been open for years.

[GeekyBear]

You do know that we are discussing something that Apple talked about doing, but never did vs. something that Google has been doing for years?

And no, this particular article about Google attempting to have innocent parents arrested was published before Apple even discussed scanning images users manually uploaded to their publicly accessible web album on iCloud.

[trvz]

Google is not Apple. Apple customers expect the higher standard.

[brokenmachine]

Well, there's a lot of things you can expect from Apple customers.

[trollbridge]

I thought the client side scanning was to protect children? If it suspects an image is bad, it blurs it and pops up a warning including a link to resources to go to for help.

Very different than trying to narc out users to the authorities.

[Zak]

There were two different technologies. One was client-side scanning for known CSAM, which created a huge backlash and is described here: https://educatedguesswork.org/posts/apple-csam-intro/

The other is detection of images that may contain nudity, whether sent or received, when the owner/admin/parent enables the feature. It is relatively uncontroversial and is described here: https://support.apple.com/en-us/105069

[tzs]

> The other is detection of images that may contain nudity, whether sent or received, when the owner/admin/parent enables the feature. It is relatively uncontroversial [...]

That's a new version. The one that as announced the same time as client side scanning to block uploading CSAM to iCloud worked like this.

1. It could be enabled on a child's device by the parents. It was not on be default.

2. If the child received a sexual image (not necessarily just CSAM...if an adult sends a dick pic to a child that is not CSAM but would have been flagged) the image is blocked, the child is notified, told their parents are worried the image may harm them, and asked if they still want to see it.

3. If the child says no, they do not want to see it, that is the end of the matter.

4. If the child says that they do want to see it and they are at least 13 they are shown the image and that is the end of the matter.

5. If the child says that they do want to see it and they are under 13, they are again told that they parents are concerned, and that if they view it their parents will be notified, and asked if they still want to view it.

6. If they say no that is the end of the matter.

7. If they say yes they see it but the parents also are notified and will be able to see it.

This should have been pretty uncontroversial, but there were objections on the grounds that if someone say sends their dick pic to your under 13 child and the child goes all the way through to step 7 and decides to view it, that is a violation of the sender's privacy because that message was only intended for the child.

[duskwuff]

One of the more recent changes is that anyone can now enable a version of this functionality, even if they aren't set up with a child account. (It's under Settings > Messages > Sensitive Content Warning.) In this mode, it behaves similarly to the over-13 mode described above; no one is notified, whether the user chooses to view the image or not.

This mode is probably best understood as "if someone texts me a dick pic, please blur it". I don't think there's any reasonable objection to this.

[ben_w]

I'm amazed it took this long for us to get male junk filters.

[Zak]

I have a hard time imagining any significant number of people objecting based on the sender's privacy. An adult sending explicit images to a child is a crime in most jurisdictions. A child doing it with a recipient under 13 is a concerning behavior the parents need to address.

Notifying the parents even requires that the child acknowledge that's what's going to happen.

[fwip]

Presumably they were worried about misclassified images, or images sent by other kids.

[themaninthedark]

The above proposed system could get ugly...

Family A enables system.

Kid B sends dick pic to Kid A.

Kid A decides to view it, Apple notifies and makes it viewable to Parents A.

Parents A are now guilty of possessing CSAM. (Distributed by Apple for extra fun)

[EmbarrassedHelp]

The controversial part is having the system enabled by default with age verification required to turn it off, and having the system impact non-Apple/Google apps. The UK for example wants Apple and Google to forcibly enable nudity blocking on all devices in the UK, and they want the system to bypass app/DRM security to scan all content visible on a device.

[simondotau]

While I didn’t agree with Apple’s approach, there is a subtle but IMHO critical nuance that is often forgotten about the client side scanning. It was only generating fingerprint material which Apple could decrypt and scan, alongside original images sent E2E encrypted to Apple servers. If the image wasn’t sent to Apple, the fingerprint wasn’t either.

And the reporting was still gated through Apple employees; there was no mechanism for the government to expand the scope without Apple’s knowledge.

I understand that people don’t like the idea of their personal hardware being complicit in treachery, but when considered from a purely functional perspective, apple’s proposal was no different to what Google and others were already doing.

[michaelmrose]

The original proposal was to use a visual hash designed to identify known bad CSAM images even if cropped or otherwise edited. Your computer would scan all your stuff for these images and report you to apple who would report you to the cops. This presented a number of issues.

Accidental false positives could lead to horrific outcomes up to and including oh look bob got shot by the cops for resisting.

It was possible to produce apparently matching innocuous images and then poison people's machines with them.Oops did you click on that picture of a tree have fun with the cops. Like an advanced form of swatting.

Although inspired by a desire to find CSAM Apple could be forced to scan for ANYTHING by repressive regimes including America and China.

Although initially targeting images client side scanning of messages is a pretty obvious next step. Again obvious good motivation exists and is completely justifiable who doesn't want to stop the next mass shooting or terrorist attack... and then we can basically use it to find people critical of the regime. Do remember we are presently prosecuting a political figure for a picture of sea shells and a guy in texas is rotting in prison for distributing political literature.

[soundnote]

> Apple could be forced to scan for ANYTHING by repressive regimes including America and China.

Including Europe. Europe is ruled by people who think 1984 was an instruction manual.

[psd1]

Europe is a fractal mix of polities ruled by a constant flux of elected representatives, and consequently exhibits political behaviour from a wide range of constituencies. Each citizen agrees, presumably, with some policies and disagrees with others, then they vote as best as they are able - which is not very, since every large electorate in the world falls short of adequate political competence. Some of those people vote for authoritarians, which i consider a shame but which is also true in your closest democratic nation.

European citizen surveillance is something i oppose, but at least it's balanced by a robust bill of human rights. America's surveillance state does not have that balance, because America wrote a bill of rights but didn't bother with the follow-through.

Kindly don't use my continent as a punching bag, thanks.

[pavon]

That is what they actually deployed. They were planning on performing client-side scanning of all images uploaded to iCloud for CSAM and reporting it to the authorities, but backpedaled after public push-back.

[ribosometronome]

The Pandora's box was already open and essentially no one noticed nor was there immense pushback that resulted in the features being removed. Photo scanning was already happening for both Android and Apple for the purpose of image search.

[inigyou]

Engineers tend to be too people-pleasing. When someone from management asks "can you scan devices for CSAM without violating privacy?" the answer should not be "hmm... well... maybe if we did it this way it could work". The answer you should give us "no" or to hedge your bets "I don't think so". Same to the government and the court. In particular a court isn't asking for a statement of objective reality whether it's possible, they're asking whether you think you could do it.

[notnullorvoid]

People pleasing is a component for some, some also like the technical challenge. The biggest factor though is that as a software engineer "No" is basically never an accepted answer (even from management with an engineering background). They'll either counter with "we need to do X, so find a way", or they'll move along the line to the next person until they get an answer they were looking for.

Edit: There are some rare competent managers who will stop the enquiry before even asking the engineers.

[mmmlinux]

Is it different than telling your therapist something in confidence and then finding police waiting for you in the lobby.

[arcticbull]

Yes, in the sense that you have a legal doctor-patient privilege that binds what they can share with whom. There's not really an Apple cloud user privilege.

No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc.

[AlexandrB]

A better analogy is a storage locker. AFAIK police need a warrant to search "your" storage locker even though it's on someone else's property. I don't see why data in the cloud should be any different. Pre-emptively scanning everyone's data is equivalent to officers rummaging through all the storage lockers in a facility "just in case" they find something illegal.

[d1sxeyes]

It’s a bit more like requiring you to submit to a weapons pat down before you go to your locker I think.

[busterarm]

> No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc.

And therapists are legally mandated to report you if you told them you viewed or possessed CSAM.

[Dylan16807]

No matter how or why? That seems like a terrible mandate.

[mothballed]

They are mandated to report child abuse. It is the same story with doctors, if an abusive parent brings in a child for care they learn never to give the kid healthcare again after the doctor reports it. It is rooted in good intentions but the effect is it means abused children never get to see doctors, therapists, get a half-ass minimal homeschool instead of going to school, etc so that mandated reporters never enter the picture.

[Dylan16807]

Reporting abuse the client was involved in has a compelling reason. That's different from hearing their client saw a picture of the abuse of a total stranger.

[busterarm]

> No matter how or why? That seems like a terrible mandate.

Honestly shocked that anyone would even say this, but even giving you the benefit of the doubt here -- the one case where I could imagine this might not happen would be if you're a police officer investigating such cases. But they also have their own therapists dedicated/trained in police-specific issues.

[Dylan16807]

Even if someone went browsing for it, yes that's illegal but there's no benefit in their therapist reporting them for just visiting terrible websites.

But also there are definitely ways to get accidentally exposed. That's an absolutely awful thing to call the cops over.

[freehorse]

That's not generally true. From [0]

> Across most states, viewing CSEM alone is generally not a mandated-reporting trigger; reporting becomes obligatory when disclosures involve an identifiable child being abused or used to produce material.

> California’s CANRA imposes a distinct duty to report electronic access (download/stream) with identifying patient information, upheld against privacy challenges based on compelling state interest.

[0] https://www.psychiatrictimes.com/view/mandatory-reporting-ch...

[FireBeyond]

I still also totally don't get their policy.

Trying to avoid false positives by not firing until a threshold was hit (was it 20 images?) seemed insane from a PR position... rightly or wrongly, all it would take would be the wrong court case and you can see the headlines:

"Apple says users can have up to 20 CSAM images on their phone before they'll tell police"

[xphos]

Imagine you have pictures of someones baptism and the kid was nude. Is it CSAM? I think the program would have to say use but morally I'd say no. The issue with client scanning is it has to assume the worst, or they are than liable. If its the person has 20+ different baptism of nude babys well huh that actually might be CSAM because the context of how that concentrated photos implies but even than its hard what if that person actually has 20 God Children its less crazy than one thinks... Especially if they have multiply phones from a single baptism.

You might not like pictures that way but honestly I think more important in procescuting CSAM is to go after the large sources of CSAM generation. Its trafficing in East Asia, and in Europe. I think weirdly America actually produces less CSAM in general because Americans are lot more off put by Sex than most other cultures. Abuse definitely happens in the US but making policy decisions like this produces bad policy.

Does iCloud rehost the photos to other people I don't really know because I use andriod tbh. If they are being rehosted (I assume to members of your contacts) that can be problematic but I think honestly the issue a lot more complex than just protect the children which the source of critic is a lot attacks against apples are coming from

[nylonstrung]

> America actually produces less CSAM in general because Americans are lot more off put by Sex than most other cultures

The US has the largest pornography industry in the world by a massive margin, and the largest consumption of online pornography per capita

Meanwhile should we be surprised that CSAM production is higher in countries like the Philippines that have very weak digital policing, abject poverty, high numbers of street children etc?

[dgellow]

The US is pretty extreme, you have at the same time easy access to all the pornography you want (unless you’re in states that require age verification), and also a very prude culture. It’s not the only place with such contradictions though

[RajT88]

Texas is particularly wild to me as a midwesterner. All the bibles and megachurches and all that and... They have strip clubs freaking everywhere! Really weird zoning laws as well - I recall seeing a strip club across the street from an Office Depot, next to a deli. The suburbs where I live, you have to go to a seedy part of town so you feel good and ashamed about it. lol

Some TV shows rightfully take aim at this contradiction (recently: The Hunting Wives).

[brokenmachine]

My uninformed guess as a non-murican would be that it's just zoned as a business.

I doubt there would be a special zoning category exclusively for strip clubs.

[dgellow]

That’s hilarious!

[mothballed]

The age of consent in the PI was like 14 until a couple years ago when they changed it to roughly match international norms, in their culture it was considered consensual rather than abuse until very recently.

[FireBeyond]

You can get married with parental or a judge consent at any age in Mississippi, New Mexico, and Oklahoma.

You can get married at 15 in Hawaii and Kansas.

You are "grandfathered" into sexual consent (not as in "marital rape" but "no longer statutory rape") when you do so.

So we may not want to be lording it too much on "international norms", particularly given that most of the marriages that happen at those ages are not "young couple got pregnant" but "older man in conservative/religious community".

[d1sxeyes]

The proposed mechanism was hash matching against known CSAM images, so the baptism photos would not trip the filter because they wouldn’t be hash matches.

[wildfireday2]

But AI algorithms actually deployed by other cloud services do not. You’d hope that they wouldn’t flag a baptism or bris photo but currently fielded systems are flagging doctor-patient medical photos and have ruined lives, so.

[dgellow]

> Imagine you have pictures of someones baptism and the kid was nude. Is it CSAM?

In some countries it is as far as I’m aware

[fortran77]

The practical problem is, without a threshold, they'd have an order of magnitude more false positives than 'real' detections, making the system useless.