BigCos, take note: you’re better off doing nefarious shit without telling anyone. Because, if you come clean, you’ll only invite an endless parade of bloggers who will misconstrue your technology to make you look bad.
No misconstrual needed. This technology is genuinely bad. It scans images against an arbitrary government-owned black-box database. There’s no guarantee that it’s only CSAM.
I have a serious problem imagining that Apple will not be willing to redeploy this technology in a novel and nefarious means in exchange for continued market access when billions are on the line.
Mainland China will probably be the first chip to fall. Can't imagine the Ministry of State Security not actively licking their lips, waiting for this functionality to arrive.
> In 1984, the U.S. Congress passed the Missing Children’s Assistance Act which established a National Resource Center and Clearinghouse on Missing and Exploited Children. The National Center for Missing & Exploited Children was designated to fulfill this role.
> On June 13, 1984, the National Center for Missing & Exploited Children was opened by President Ronald Reagan in a White House Ceremony. The national 24-hour toll-free missing children’s hotline 1-800-THE-LOST opened as well.
> The Missing Children's Assistance Reauthorization Act of 2013 (H.R. 3092) is a bill that was introduced into the United States House of Representatives during the 113th United States Congress. The Missing Children's Assistance Reauthorization Act of 2013 reauthorizes the Missing Children's Assistance Act and authorizes $40 million a year to fund the National Center for Missing and Exploited Children.
They're not owned by the federal government, but they do get a lot of federal government money.
> The National Center for Missing & Exploited Children® was established in 1984 as a private, nonprofit 501(c)(3) organization. Today, NCMEC performs the following 15 specific programs of work, funded in part by federal grants (34 U.S.C. § 11293):
Source: https://www.missingkids.org/footer/about
That’s like saying the federal reserve is “private”. No, the NECMEC is not a private entity. Not only was it heavily funded/created by the gov, but more importantly it is granted special legal status. You and I can’t just spin up our own CSAM database. Nor do we have any laws that say that any companies aware of CSAM must send it to us and only us.
It's important to keep nefarious stuff on the server side because eventually someone will reverse engineer what's on the client side.
Imagine if Apple had done this on the client side without telling anyone, and later it was discovered. I think things would be a whole worse for Apple in that case.
Devices with proprietary OSes spend more and more time phoning home, then exchanging data officially for "updates". They probably tell the truth, but should one of them decide to hide users data exfiltration or other monitoring practices behind those updates, it would be quite hard to catch them. In other words we have no way to tell that they're not already doing this.