[gabrielblack]

My opinion about: 1. Every pedophile know about the existence of this system, so I don't think it will be useful to fight those monster, maybe only marginally;

2. Anyway, is that legal ? Even if some crazy store material on his Apple hardware isn't that illegal search non usable in law courts ?

3. Child abuse is often used as Trojan horse to introduce questionable practice. What if:

- the system is used to looking for dissidents: I look for people that have a photo of Tiananmen Square protests on their pc, for example;

- for espionage: I have the hash of some documents of interest, so all the PCs with that kind of documents could be a valuable target;

- profiling people: you have computer virus sample on your PC -> security researcher/hacker;

I think that the system is prone to all kind of privacy abuse.

4. this could be part of the previous point, but, because I think it's the final and real reason for the existence of that system, I give to this point its own section: piracy fight. I think that the one of the real reason is to discourage the exchange of illigal multimedia material to enforce copyrighs.

For the listed reasons, I think that is a bad idea. Let me know what are you thinking about.

[larschdk]

5. The system can easily be abused by governments or malicious actors to frame innocent people. People merely suspected of keeping such images are de-facto punished and stripped of rights even without standing before a judge or getting a conviction.

[newbamboo]

This is my primary concern. It will become a weapon to destroy the lives of anyone who is targeted by someone with middling or better hacking skills. A sort of digital “swatting” that makes using apple products a no go for anyone with cyber-enemies (one can’t opt out of Apple ID and apply security updates).

[GeekyBear]

Google has been scanning your entire account for kiddie porn for the past decade.

>a man [was] arrested on child pornography charges, after Google tipped off authorities about illegal images found in the Houston suspect's Gmail account

https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-le...

Their system can easily be abused by governments or malicious actors to frame innocent people.

[Unklejoe]

Once again, this scan takes place on _their_ servers on data that is stored on _their_ servers. It does not take place on the device itself, which is the case with this new Apple thing.

[GeekyBear]

Once again, the notion that everything on the device is scanned under Apple's system was never true.

Only photos you attempt to upload to Apple's iCloud are scanned. If you turn off iCloud photos, NOTHING is scanned.

>Q: So if iCloud Photos is disabled, the system does not work, which is the public language in the FAQ. I just wanted to ask specifically, when you disable iCloud Photos, does this system continue to create hashes of your photos on device, or is it completely inactive at that point?

A: If users are not using iCloud Photos, NeuralHash will not run

https://techcrunch.com/2021/08/10/interview-apples-head-of-p...

[Unklejoe]

> Once again, the notion that everything on the device is scanned under Apple's system was never true.

That isn't what I said.

Also, that's not why most people are so upset. Most people are so upset mainly because Apple has now proven that the capability exists, so they can now be more easily compelled by governments to scan for "extra things".

Prior to this, if a government asked Apple to scan someone's phone, Apple could respond with "we don't have that capability", and it would presumably be a tough legal battle to force a company to add a capability that doesn't exist.

This hurdle is now much lower. The effort has gone from "force Apple to design a new system for scanning phones" to "add these couple of hashes to the pre-existing database".

Also, expanding this from just iCloud upload candidates to the entire device is a very small leap now. I mean, the bad guys could just turn off iCloud, and we must think of the children...

Then you have Apple's "reassurance" that they won't comply with government requests to scan for additional things, which is completely moot considering Apple relies on a third party database and has absolutely no control or idea of what the hashes really are.

[GeekyBear]

The notion that scanning cloud data on device is somehow worse than doing the same thing on server is deeply flawed.

If you have a false positive on device, nothing is sent to Apple's servers. It takes several (possibly false) positives at once to trigger a human review.

If you have a single false positive on server, that data is sitting there where it can be subpoenaed and abused.

Also, recent history shows that Apple is willing to fight government demands to invade user privacy in court.

[lucasyvas]

This fact is well known and changes nothing. The problem is that the system exists at all. The fine print WILL change - it always does, and that's also a well known fact.

[GeekyBear]

By that logic, Google will definitely begin selling their treasure trove of user data to anyone with a checkbook, because the fine print WILL change.

[csmiller]

It’s a difference in policy vs technical capability. Currently the policy is only scan when iCloud Photos is enabled, but the capability to scan at any/all times is just a policy change away.

[GeekyBear]

No, it's a difference between scanning the files that users store in their respective clouds on-server or on-device.

Scanning on-device (where a single false positive cannot be subpoenaed and misused to incriminate their customers) is simply more private.

>Innocent man, 23, sues Arizona police for $1.5million after being arrested for murder and jailed for six days when Google's GPS tracker wrongly placed him at the scene of the 2018 crime

https://www.dailymail.co.uk/news/article-7897319/Police-arre...

[adventured]

Apple's new system is scanning personal property that doesn't belong to them and isn't yet in their cloud.

Gmail files that get scanned are contained on Google's property, in their cloud, on their machines.

Entirely different context.

It's the difference between the USPS coming into my home without permission and going through my documents, records, mail - versus if I send mail through their system and they track it, scan the envelope, etc.

The iPhone used to be pretty obviously personal property, now Apple is saying that's clearly no longer going to be the case going forward.

[crusty]

Oh, I like this USPS analogy but I'll clean it up. Google photos and chat are like a USPS that only stores and transmits post cards. It's understood by the creator/sender that anyone who has access to them can read them. Apple here is a USPS that sends sealed envelopes. They (say they) can't read what's inside as it's sent or stored. With this change they will create the 'capability' to show up whenever you decide to send an envelope and read it before you seal it up for sending.

Meh, nevermind. That's not much cleaner.

[GeekyBear]

> Apple's new system is scanning personal property that doesn't belong to them and isn't yet in their cloud.

Apple's new system only scans photos you attempt to upload to their cloud.

Nothing else is scanned.

Scanning the files on server, the way Google and Microsoft do it, means that false positive data is lying around where it can be subpoenaed and used to incriminate innocent people.

>Innocent man, 23, sues Arizona police for $1.5million after being arrested for murder and jailed for six days when Google's GPS tracker wrongly placed him at the scene of the 2018 crime

https://www.dailymail.co.uk/news/article-7897319/Police-arre...

[_-david-_]

>Apple's new system only scans photos you attempt to upload to their cloud.

And what if in the future they decide they need to scan more than images going to the cloud? What if there is some huge epidemic of child abuse or some other terrible thing and Apple decides they need to do more?

Once you open Pandora's Box you can't close it.

[GeekyBear]

What if in the future Google starts selling your location data to anyone willing to write them a check?

Once you open the Pandora's Box of collecting location data, you can't close it.

[arvinsim]

It doesn't matter if the scan is conditional.

It matters that the capability is there.

[nicce]

Capability has always been there. It is only worded now in a way which made it the most people understand. Speculation about "doing something in hidden", is as valid as before.

In reality, we can be only be mad when they they are publicly making things worse in black box systems. Not about something, which is "policy change" away. Let's be mad when they actually change that policy.

[strofcon]

Your whatboutism is profoundly unhelpful here.

Yes, other companies are doing bad things, and they should be stopped.

Doesn't by any stretch of the imagination mean that Apple should be allowed to do something even worse.

[GeekyBear]

Other companies aren't "doing bad things" they are handling scanning the contents of their cloud services in a much more user hostile way.

Keeping that data on their server means it can be subpoenaed and misused.

[eplanit]

Today they look for child porn, tomorrow it will be "Covid Misinformation" (which is today's terrorism). A blind man with a stick can see it coming.

[docmars]

I can't reply to or upvote ~stalkersyndrome's response to add my applause, but I'll do it here instead. He's right. It's inconvenient, and people don't like reading that, which is why it's downvoted to dead, but alas, this is the same reason people avoid getting counseling, because they haven't worked up the courage to be honest with themselves yet. We'll get there, I hope.

https://news.ycombinator.com/item?id=28156934

[lovelyviking]

Perhaps we should also rise again the question about who controls own computer because any abuse starts there.

Only full control over own device can prevent abuses. Especially when device comes any close to definition of being personal. You should be able to install own software on the personal device. Including os and bios/firmware.

[villgax]

*laughs in Minix land

[_trampeltier]

An inside joke :-)

[Bodell]

I just want to add to your number 1.

Not only would this be marginal it also wouldn’t necessarily be catching the real “monsters”. I don’t think if you find someone with old already known about images that it would necessarily equate to someone that actually abuses children. I think about this in a similar way (not exactly) as I do with drugs, just because a person gets busted with drugs doesn’t mean they are a drug dealer or a maker of drugs.

This is not to say that perhaps there are some more active real-time stuff in these databases that maybe with enough searching could make its way back to the perpetrator and indeed maybe even find a victim. It’s just seems that that would be far more marginal and is generally what I’m concerned about when it comes to these issues. For me it’s more important to protect children than it is to bust some weirdos for looking at the wrong porn (these can both be related as well and I do understand that I just think it’s not as cut and dry as we believe it is), further if it keeps said weirdo from actually harming a child then let them have it. We allow these databases to exist for, presumably, the same reason, with the idea that we can stop future victims from happening.

[tsherr]

As I understand it, in Canada, having a picture of your child in a bathtub is child porn. So I can see this going horribly wrong.

[swiley]

The Canadian government has really lost their minds, that's where ignoring principles gets you.

[3pt14159]

Citation needed. Many parents here have photos of their kids in bathtubs.

[jackson1way]

Many people steal bicycles and are not in handcuffs or even prosecuted. Doesn‘t mean it‘s legal, right?

[newbamboo]

Selective enforcement should concern anyone who cares about civil liberties.

[3pt14159]

I still don't see a citation, and I'm not exactly about to google for one and click around.

[nzealand]

> 2. Anyway, is that legal ? Even if some crazy store material on his Apple hardware isn't that illegal search non usable in law courts ?

Yes, it's considered legal. Apple reviews the content first. Courts say this means it is not an illegal government search. It's a search by a private party, who then manually decides to notify the government.

[cookiengineer]

> Yes, it's considered legal.

No, it's not. At least not here in Germany. By law, even police officers are not allowed to look at child porn. The only institution explicitely allowed to do so is the BSI.

The rest of the population implicitely incriminates themselves when they look at (not own) child porn, including Apple's legal entity or employees.

See [1] for 184b Strafgesetzbuch

I'm trying to point out that with this action Apple bluntly decided to ignore a whole lot of countries and their federal laws, which is not something I would embrace - even when they had good intentions.

[1] https://www.gesetze-im-internet.de/stgb/__184b.html

[nzealand]

I think they are only planning to scan US phones, so I am not sure how German law applies here.

[janeroe]

> What if: the system is used to looking for dissidents ... / espionage ... / profiling ...

Not if. When.

[zionic]

Lets go one step further, Apple is a content producer with multiple streaming services now.

Using this system to look for unlicensed content will be irresistible to them.

[aborsy]

I agree!

[snarf21]

I think this is a bad move by Apple even if the point is to set up E2EE later. However, one thing that everyone seems to forget is that all these pictures were already being sent un-encrypted to iCloud. ALL of the same issues already completely exist today and were already being scanned and we have heard no outcry. ALL of the same loopholes and unreasonable warrants can be used against you today with all of the un-encrypted data they have on their server right now.

The one thing that occurred to me is that this is almost seems like this is a cya, Section 230 protection in disguise. There has been more discussions about Big Tech and 230, and this is one way to say "Look, we are compliant on our platform. Don't remove our protections or break us up, we are your friend!" It also shouldn't be too surprising given how Apple has behaved in China. They will only push back against the government up until the point it starts to affect profits.

[jackTheMan]

Even if it is possible in other ways as well today, this is a black pattern of going down step-by-step.

When it will be when people will say no? These are all small steps only.

[nicce]

> this is a black pattern of going down step-by-step.

This is very hard to argue. Functionality like spying all your files is trivial to add, and technically we haven't really moved anywhere. "Now technology is there", is not valid argument since it has always been there. Scanning your files and send some metadata is the feature which requires least effort to make from everything that Apple has released.

It might feel bad, when your device scans your to-be-uploaded-cloud images now, but iOS has never been yours. It is very closed system, a part of Apple-ecosystem. Only a guy who as has access for whole iOS source code knows what is actually happening in there. On Apple-ecosystem, only the final result matters in reality and what they say. Since your device is not really yours, you should think like that you are just using Apple-ecosystem, being part of it. If you don't want that, you should have switched into some Linux phone already.

You can speculate all-day what else it might do in hidden in the future. Speculation about hidden features is as valid now as it was yesterday or will be tomorrow.

In reality, we can only be really worried when they publicly say something, which finally makes the end results worse. This did not happen yet. Actually the opposite happened, but here we go.

We have been trusting Apple for quite some time, and they really haven't got caught on doing something else than they have said, so what has changed?

[strofcon]

And Google's entire Code of Conduct used to be "Don't Be Evil". Things change. Money drives all decisions, at all levels above the visceral.

Unfortunately, historical precedent for any given business entity provides absolute zero evidence of probable future behavior. :-\

[nicce]

Shouldn't we change our behavior after the bad things happen? Not based on speculation? I have dumped Google completely once they changed drastically.

[strofcon]

I agree we should do so! That said, we should discourage behavior that is absolutely guaranteed to be abusive in the future.

We can call it speculation - that doesn't mean it's wrong. Power is the play, and companies will always be leveraged for the benefit of the powerful. This seems pretty indisputable to me.

[snarf21]

I agree and my first statement was this is bad. My point was why has no one been complaining about it already being very bad and reacting with "I'm selling all iDevices" when it gets worse?

[simonh]

Of course it would be possible to implement content search, profiling and reporting mechanisms for such content, but this seems to be a singularly bad platform for that sort of search.

The image profiles are part of the OS so there's no mechanism to deliver image profiles separately for different countries. Also when the threshold number of matching images is reached, the matches are reported to a manual reviewer at Apple not a government. It only checks images on upload to iCloud photo storage.

So of course each of these limitations of the system could be changed, but you'd really need to change all of them and at that point you've created a completely different system. There's no simple change to this system that would suddenly turn it into a snitch for e.g. China or Saudi Arabia.

I've seen exactly the same objections raised every time any kind of device content search has become mainstream. Back in the 90s it was virus checking (Do you trust the AV company? What if they were bribed by the content companies?), full device indexing and search (Do you trust the OS vendor? What if they're in league with the government?). I'm very surprised this didn't blow up when Apple implemented ubiquitous image text recognition. Maybe it did. AV and device indexing mechanisms, which are ubiquitous, seem like a far more vulnerable target for such requirements.

So I don't really buy the slippery slope argument. In theory any government could pass a law requiring any company operating in it's jurisdiction to do anything, with an implementation suitable to that actual purpose. Of course this mechanism is motivated by laws in the US so it's a perfect example of exactly that, and it's a completely new system not a slippery slope subversion of an existing one. The real slippery slope here is legislative, not technical and I think that should be far, far more concerning.

I do think the legal and moral questions about this mechanism are legitimate. I think it would make more sense for Apple to scan photos in their cloud storage on the cloud storage rather than on upload. I understand there are theoretical privacy benefits to users from this implementation but the optics of having user's devices snitch on them are all wrong.

[luckyandroid]

>Back in the 90s it was virus checking (Do you trust the AV company? What if they were bribed by the content companies?), full device indexing and search (Do you trust the OS vendor? What if they're in league with the government?)

These are examples of companies choosing to do something as a selling point of their software as a benefit to the end user, and people worrying that it could aid the government down the line if they change their mind.

Apple's content review change is explicitly FOR reporting people to police in a way that can be expanded beyond it's currently set purpose (child porn) later.

>I'm very surprised this didn't blow up when Apple implemented ubiquitous image text recognition.

I'm personally not a fan of that stuff anyway, but personally if it's only my local device I don't tend to care about image recognition, it's only when it involves communicating information from MY hardware to THEIR servers that I get antsy.

[simonh]

>Apple's content review change is explicitly FOR reporting people to police in a way that can be expanded beyond it's currently set purpose (child porn) later.

I think it would be very hard to expand this beyond it's currently intended purpose, for the reasons I've given. It's terrible for identifying dissidents because it only catches them if they upload to iCloud servers. Dissidents are much more likely to be tech savvy than random child molesters. The reports have to go through Apple, and don't go directly to the cops. Also it's a global image profile list so it's not possible to keep country specific updates secret.

An effctive surveillance mechanism would need to change all of these.

[Nicksil]

>It's terrible for identifying dissidents because it only catches them if they upload to iCloud servers.

This is a configuration change. Without knowing the implementation, I'd bet a lunch that, for the time-being, the reason this thing is executed only upon upload to iCloud is because there's some simple business logic buried in there telling it to do so.

>Dissidents are much more likely to be tech savvy than random child molesters.

This is a curious argument. You didn't explain why you think this might be. What is it about a dissident that makes him or her more savvy than some random child molester?

>An effctive surveillance mechanism would need to change all of these.

If true, the obstacles you outlined are trivial to overcome.

[simonh]

>This is a configuration change.

Not it isn't, the check is built into the upload client, they'd have to implement an on-device storage scanning mechanism. That's a different type of system implemented in a different kind of service.

Not that doing that is hard at all, it's not rocket science and they already have full-system indexing and search, but that's also why this isn't a significant step down any kind of technical slippery slope. The problem here is legislative, not technical.

[aNoob7000]

Apple should just scan the pictures that are in iCloud (their servers). They just assumed that if you have the iCloud option enabled on your device that it gave them the right to do the scan on your phone/computer.

I want to also point out that A/V companies never said they were going to scan for child abuse images on your computer and report you if they found any.

Like you said, the optics are terrible.

[nicce]

> Apple should just scan the pictures that are in iCloud (their servers). They just assumed that if you have the iCloud option enabled on your device that it gave them the right to do the scan on your phone/computer.

End result is the same. Difference is, that now Apple has very limited access to your images. You can only trust in closed systems. When you step into the Apple ecosystem, you are giving a lot of trust.

> I want to also point out that A/V companies never said they were going to scan for child abuse images on your computer and report you if they found any.

Why would they say, if it is perfectly legal to do anyway. They literally scan every file, so no need to mention anything specific which could lead only for negative PR.

[zionic]

Apple has unlimited access now, all that stands in their way is the thinnest of policy lines.

[nicce]

> Apple has unlimited access now

Always been. You don't own your iOS based device which is very closed source and mostly unusable for any other operating systems.

[LatteLazy]

>The image profiles are part of the OS so there's no mechanism to deliver image profiles separately for different countries

Haven't Apple already said it WILL be country specific?

>Apple’s new feature for detection of Child Sexual Abuse Material (CSAM) content in iCloud Photos will launch first in the United States, as 9to5Mac reported yesterday. Apple confirmed today, however, that any expansion outside of the United States will occur on a country-by-country basis depending on local laws and regulations.

https://9to5mac.com/2021/08/06/apple-says-any-expansion-of-c...

I think they'd need to be country-aware at least, otherwise the FBI or whoever will get reports for all people on earth when they presumably don't need them for anyone outside the US?

[simonh]

Reporting is country specific and US only yes, but the profiles are delivered baked into the OS. I suspect this is so that pedophiles can't buy a phone mail order from Canada and bypass the system.

[LatteLazy]

I think the profiles will need to be country specific too. What counts as CSAM in some places doesnt in others (here in the UK we have a ban on cartoons but bath pics are allowed for instance).

This is something Apple have been pressed on a lot. So far (I'd be happy to be corrected) they've only said "whatever local law permits". That sounds ok, till you realise Saudi will want gays reported and China wont like any Winnie the Pooh pics...

[simonh]

China already operates their own iCud storage so this is irrelevant to them.

Apple doesn't have any iCloud data centres in Saudi, so Saudi can't pass laws about what is or isn't stored in them.

Look, the way this works and how it's implemented matters. It's stunning to me how many people are thoroughly confused and jump to unwarranted conclusions about how this actually works and what that means.

[LatteLazy]

I dont think your saudi or china points grasp the nature of this tech. This is about checking what users have on their devices BEFORE it is uploaded to icloud.

So both China and the Saudis (any plenty of other governments) will be very interested as right now, it takes a lot more effort for them to access phone contents (there certainly aren't mass surveillance programs like this for handsets).

I weirdly agree with your last paragraph, but i think we disagree about the details. I can't find any evidence for your assessment that this can only be used against 1 (US) set of image hashes. Or that shitty regimes won't be allowed to abuse it.

If Apple came out and proved that, i might not be happy but my worst fears would be gone. Their silence is sort of deafening at this point...