[foxyv]

The thing that scares me isn't so much the violation of privacy. It's the idea that some computer algorithm can accuse me of a crime automatically with no evidence and generate an investigation.

Judging by how police respond to these leads, you can end up in jail based on this "Evidence." While you wait for a 6 month investigation to be completed you lose your job and get an arrest record. Even if your photo is just a picture of static on a TV which produced a false positive.

It reminds me of the dog who always indicates the presence of drugs 100% of the time. Probable cause, made to order.

[aNoob7000]

I want to add that they are scanning your property without your permission. The crazy part is that the government can't do this with a warrant or probable cause, but Apple decided that it can and will.

Amazing that Apple destroyed their image about "Privacy" in minutes.

[api]

I really wonder about their motives for this. It really has destroyed their image to the point that I'm thinking about abandoning the platform if nothing changes.

This will not just be about CP. In many countries this will instantly be used to detect any kind of dissent. In the US it will take longer, but eventually I can imagine "misinformation" as determined by some stupid algorithm flagging you for quiet social credit style lists. What constitutes misinformation will change with the political winds.

Hey... maybe we can let people select which reality they wish to be enforced! Maybe Republicans can get flagged if they spread memes about universal health care while Democrats can be flagged for discussing LGBT rights!

The only thing I can imagine is that Apple is facing enormous pressure from behind the scenes to become part of the panopticon. Private companies can't really resist governments.

[heavyset_go]

> This will not just be about CP. In many countries this will instantly be used to detect any kind of dissent. In the US it will take longer, but eventually I can imagine "misinformation" as determined by some stupid algorithm flagging you for quiet social credit style lists. What constitutes misinformation will change with the political winds.

In the US, there are over 70,000 overdose deaths a year[1], and that number has increased dramatically each year.

If Apple is already scanning photos and messages, why not save 70,000+ lives while they're at it by detecting heroin and fentanyl dealers, too?

[1] https://www.cdc.gov/drugoverdose/deaths/index.html

[oceanghost]

What I wonder is-- are child pornographers really by and large storing these materials on iCloud or on their phones at all?

I think you're correct, this is the thin end of the wedge for something else entirely.

[api]

This is the thin end of a wedge even if it isn't intended to be. Rights need to be strictly defined for a reason.

I personally believe that we had an attempted fascist coup in the USA in January of this year. I know a lot of people disagree. If you disagree imagine that the defeated POTUS had been someone to the left of Bernie Sanders and a bunch of militia members wearing "Lenin was Right" shirts and carrying nooses and zip ties had stormed the capitol to stop the vote from being counted. What would you call that? (I'd call it the same thing.)

The next attempted coup will not be so incompetent. That totalitarian regime will inherit everything we are building today.

If you're on the other side of the aisle, imagine that the next attempted coup is indeed a radical leftist. That's by no means impossible. If the economic divide continues to worsen in this country it may not matter if they are "right" or "left" as long as they are passing out torches and pitchforks. I could see a large fraction of the MAGA crowd turning red in the more classic commie sense if they were promised jobs, affordable housing, and... well... someone to make America great again.

[Scaevolus]

Not that anyone reads them, but every cloud storage EULA has a clause describing how they may remove content at will, especially if it is illegal. That's their justification for why this scanning is happening on photos that are being uploaded to iCloud, and not simply all photos on every device.

https://www.apple.com/legal/internet-services/icloud/

> Apple reserves the right at all times to determine whether Content is appropriate and in compliance with this Agreement, and may screen, move, refuse, modify and/or remove Content at any time, without prior notice and in its sole discretion, if such Content is found to be in violation of this Agreement or is otherwise objectionable.

[aNoob7000]

I have no problem with Apple, Google, Microsoft, etc.. Removing and reporting illegal content from their servers. My issue is with Apple doing it on your device or machine.

When did Apple become law enforcement?

[zepto]

> I want to add that they are scanning your property without your permission.

This is a false statement. Scanning is opt-in.

[salawat]

Allow it or don't buy our product is a fairly disingenuous take on "opt-in".

[zepto]

> Allow it or don't buy our product is a fairly disingenuous take on "opt-in".

That’s not how it works.

The only thing it affects is iCloud Photo Library. There are numerous competitors you can choose from if you want a cloud photo library, and nothing prevents you from just keeping your photos on device or exporting them to a PC or Mac.

[jrm4]

Say it louder. Recently I opposed this and was met with "I bet you don't have children."

I do, and I am black and so are they, and I'm quite familiar of the present state of facial recognition tech (and, of course, the justice system generally) when it comes to people who look like us.

This is all unacceptable, and the fact that some of it has already come to pass is no reason to not fix it.

[zepto]

What does any of this have to do with facial recognition tech or race?

[jrm4]

I get that questions like these are perhaps meant to be "more precise" about the specific tech in question, but honestly, they do more to display the naivete of the questioner in terms of understanding how this sort of thing plays out in real life, when it's not just tech folks, but other policymakers and stakeholders are involved and have the power to make decisions.

Even if the thing Apple is talking about right now can be distinguished from "facial recognition" on a technical level, it would be much MORE of a mistake to NOT lump them in together if we're trying to bring this debate to the general public, which we, of course, we should.

(don't get me started on race..)

[zepto]

> it would be much MORE of a mistake to NOT lump them in together if we're trying to bring this debate to the general public, which we, of course, we should.

Frankly this suggests that you think it’s a good idea to mislead the general public. I think that is one of the ways we harm our public discourse. I could be missing a connection between the two that is obvious to you but not to me, in which case I apologize.

I am not naïve about facial recognition. I am not white, and I have known about ML technology since the 90s. Long before it became a known problem it was obvious to me that ML models would end up simply reflecting the biases of the corpus they were trained on, and this would lead to them embodying discrimination of one kind or another, some of which would not be obvious in advance. The perceived ‘neutrality’ of algorithms would be and in fact is invoked to minimize this problem, when of course the problem is not with the algorithms but with what people feed into them.

Please let me know if that doesn’t capture the problem with facial recognition adequately.

So, given that I’m not naïve about facial recognition, I ask again - what is the connection you see here between racially biased facial recognition and Apple’s CSAM countermeasures?

[jrm4]

Not at all -- what I'm saying is that tech people end up misleading themselves in terms of likely outcomes by having these discussions and focusing on the hard discrete lines around this or that particular technology.

What very reliably happens is -- thhe people who make the decisions (who, unfortunately, are very rarely tech people) will lump them in anyway LATER, when it actually matters and is too late.

So your point is technically correct, and simultaneously absolutely does not capture the problem with facial recognition adequately, because it doesn't factor in "if you get people to sign off on Apples specific thing today, you'll basically be able to sign off on just about anything that sort of looks like it to the layperson" tomorrow.

[zepto]

Well the ‘tech people’ line is an ad-hominem. I don’t see anyone doing a great job of knowing the right move in terms of communicating about complex issue to the public. Aren’t you a tech person too?

Who do you see as ‘the people who make the decisions’ in this case? I.e. who do you imagine will lump these together? I could see the FBI saying ‘you did CSAM detection, so surely you can do ‘child predator detection via facial recognition’. Is that what you mean?

As for getting people to sign off on stuff - I think it’s not so obvious what we do and don’t want people to sign off on. Not implementing something like this now could easily mean they are forced to scan in the cloud, and that really would be a slippery slope.

It seems like you are saying even though this thing isn’t bad, we should persuade people into not signing off on it because we want to make sure they don’t later sign off on some facial recognition thing that actually is bad. Is that an accurate enough paraphrase?

[heavyset_go]

This is why algorithmic and predictive policing is so terrifying. Everyone in the justice system now has a great way to shirk liability for their actions and decisions: they can just defer to the infallible black box system that tells them who is a criminal or not. Want to side step accusations of bias or targeting? They were just going after whoever the black box said was guilty, and computers purportedly are not biased.

The corollary to this shirking of responsibility is how it lets people, and not just law enforcement, justify abuse of whoever the system says is guilty of a crime, because after all, the company that made the black box says that there is a one in a trillion chance of encountering a false positive. Judges will throw the book at defendants because, statistically, the black box system is almost never wrong, and the system says the defendants are monsters.

But the most Kafkaesque part is that people will never get an answer for how these systems determined they were guilty. It's a trade secret, it's part of on-going investigations, it's critical to national security, or in Apple's case, it's literally illegal to look and find out how their system came to conclusion because viewing the data itself is a crime. These systems often inscrutable ML models, as well, and we all know just how buggy and error prone computers and software can be.

[zepto]

> The thing that scares me isn't so much the violation of privacy. It's the idea that some computer algorithm can accuse me of a crime automatically with no evidence and generate an investigation.

Nothing about apple’s proposal involves algorithms accusing people with a crime without evidence.

All actual alerts are done by humans checking the photos.

[foxyv]

Predictive policing models require a human in the loop as well. However people tend to trust these algorithms far beyond their reliability without knowing how they work. As I understand it, human reviewers for this program do not see the photo itself, but instead see the hash and make a determination from that.

A positive result may be enough to see all of your devices in evidence bags for the next 6 months and serve as probable cause for warrants. In addition, how hard is it to subpoena Apple for a report on how many times a whistleblower's device has been flagged then use it as probable cause. Or, cherry pick which reports Apple makes to target undesirables.

[zepto]

> As I understand it, human reviewers for this program do not see the photo itself, but instead see the hash and make a determination from that.

No, this is not correct. Human reviewers see a visual derivative which is separate from the hash. It’s basically a blurred thumbnail - enough to visually confirm that the image is not a false positive, buy not enough that the reviewers are constantly exposed fo child porn.

Also remember that multiple matches are required to even get to the human review.

The rest of your comment really doesn’t seem to match the system being described. It’s not predictive policing or anything like it, and it is obviously very much against Apple’s interest for it to generate false positives.

[foxyv]

> It’s not predictive policing or anything like it, and it is obviously very much against Apple’s interest for it to generate false positives.

It is not predictive policing. However it's a pretty close cousin. Automated policing. I'm also skeptical that a blurred image will be enough to confirm/deny CP. I'm pretty sure that it's a system similar to YouTube's content ID and will work out in a very similar fashion. Also they have a very good incentive to err on the side of false positives in order to reduce liability for hosting CP on their servers.

It worked for the DMCA, now law enforcement are trying something similar for CP.

[zepto]

> It is not predictive policing. However it's a pretty close cousin. Automated policing.

It’s not policing. This is a mechanism to detect if people are uploading known child pornography images to Apple’s servers without giving Apple access to your photos. That is the only use case for this system.

Yes, if you try to upload such a collection, a police report may be filed, but only if you do this specific thing and it is verified by humans.

> I'm also skeptical that a blurred image will be enough to confirm/deny CP

It doesn’t have to confirm CP - it only has to confirm that the image matches the known CP from the database.

[smsm42]

I'm not buying it. If you can recognize what it is, enough to be able to be sure, you're exposed to it. If not, you're just guessing - and your guess is heavily biased towards confirming what the machine said, because Apple spend $MASSIVE_AMOUNT on this technology, and who are you to question it based on a blurry picture? Also, you'd be saving children, and if you're wrong (which you're likely not - remember, industry-leading AI!) - well, the police surely will find that out very quickly and everything will be fine.

It's not like it's the first time such systems are built. We have a cases of Big Social banning people for random stuff and then saying "it was a technical error" when the noise in the media is strong enough. We have chess channels banned for racist hatespeech. Only this time the question is not whether you will be denied the opportunity to post cat pictures on facewitter for a week. It's pretty much the most shameful accusation one can be subjected to in our society. Once the press gets hold of it - and it'd get hold of it the minute the police does - there's no coming back from it for the person affected (well, maybe if they are Hunter Biden, but not otherwise). And all that will be hinged on an anonymous drone looking at a blurry picture?

[zepto]

> I'm not buying it. If you can recognize what it is, enough to be able to be sure, you're exposed to it. If not, you're just guessing

That’s a misunderstanding of what is happening. The reviewer doesn’t have to see whether the visual derivative looks like child porn. They only have to see whether it matches the visual derivative of the child porn image that the hash matched.

[smsm42]

So they'd just compare two small very blurry thumbnails? Not sure that makes it any better.

[throw03172019]

There is a human review (ugh.) as well.

But still, your example is very concerning.

[falcolas]

From Apple's announcements, there is a human comparison of the visual hashes. They don't appear to be forcing humans to view the images themselves.

[y7]

The technical assessments speak about "visual derivatives". I'm guessing they mean a low resolution thumbnail.

[foxyv]

They say there is a human review, but fail to explain the parameters of the review.

How many photos is a reviewer expected to review in an hour? What are their incentives and performance metrics? Do they have a 3 party voting system to confirm? For positive human review is there additional scrutiny? Is data anonymized properly? Does law enforcement receive a copy? Is this going to be limited to child exploitation?

Without this info I just have to assume it's some guy in a box expected to review 1000 pictures an hour and gets scrutinized if he doesn't click at least 10% positive. So he's just counting 1 positive 9 negatives.

[gjsman-1000]

Remember they said the odds of an account being falsely reported were "1 in a trillion."

We have no math to back that, some have called it BS. However... that would mean that one person could easily review only a few photos a week even if it was way off.

[throw03172019]

I would hope they aren’t incentivized to incriminate innocent people.

[foxyv]

I would hope so as well. However, my hopes for these policing algorithms have been dashed too many times. The potential for abuse is just too great without effective laws and appropriate procedures. We've seen the results at YouTube, Pasco County Florida, the UK, and Facebook.

There is a reason that a key part of justice is being able to face your accusers. Secret algorithms are being used to abrogate this right. You can't cross examine an algorithm. Expert witnesses to contradict its expertise are too expensive for most defendants. Then the developers of the algorithm can claim with plausible deniability that the algorithm is 1 in a trillion accurate.

It's essentially the new bite mark analysis and polygraph pseudoscience used to elicit false admissions of guilt or convince gullible juries. "We have the hash of your photo George, we KNOW you are a pedophile. Just admit it and you can go home. We'll go easy on you. We promise!"

https://www.forbes.com/sites/nicksibilla/2021/04/26/lawsuit-...