Meta cancels the contract with the outsourcing company they contracted to classify smart glasses content after employees at the company whistleblow about serious privacy issues with the content they were paid to classify.
How else do you want companies to remove and prevent CSAM? It seems like you must have some human involvement to train and monitor.
It’s a terrible job, I wouldn’t want to do it, but someone needs to. Perhaps one day, AI will be accurate enough to not need it, but even then you need someone to process complaints and waivers (like someone’s home photos being inaccurately flagged).
> How else do you want companies to remove and prevent CSAM?
Different situation.
Facebook has to do CSAM moderation because it's a publishing platform. People will post CSAM on facebook, so they must do moderation.
And "just don't have facebook" isn't a solution because every publication of any sort has to deal with this problem; Any newspaper accepting mail has this problem. (Albeit to a much more scaled down version) People were nailing obscene things to bulletin boards for all recorded history.
---
In contrast, OpenAI has no such problem. It did not have CSAM pushed onto it, it actively collected such data itself. It could have, at any point before and after, simply stopped scraping all of the web indiscriminately and switched to using more curated sources of scraped data.
The downside would be "worse LLMs" or "LLMs being created later", which is a perfectly acceptable compromise.
---
This is not to say that genuine content flagging firms have no reason to curate such data & build tools to automatically flag content before human moderators have to. (But then they also shouldn't be outsourcing this and traumatizing contract workers for $2-3 an hour)
But OpenAI is not such a firm. It's a general AI company.
There's no dollar amount but proper support during and after employment is a minimum, and a large paycheque will both offset some of the human cost and make it easier for people to be pushed to quit the job; Such that they aren't doing the job for too long.
The current support systems for police in this subject are already insufficient. Facebook's treatment of their moderation staff is abhorrent. The point of including the pay figure is to further illustrate just how damning this subcontracting practice is.
There is labor that is necessary for our societies to function, but a direct threat to the people doing the work. Someone has to do it, and it should be seen as a great service to society and rewarded accordingly. In a just world, we would be paying significantly extra for threats to health that come from work, in the one we are currently in we use threat of worse harm instead.
Someone has to do it, and it should be seen as a great service to society and rewarded accordingly
You are just too priviledge to understand people: many people would be glad do do it for the minimum wage, I would fight to have that oportunity (I live in west EU).
We have coal miners destroying their bodies and lungs, cobalt mining slavery, cocoa nut child labour and de facto slavery, sex workers, CPS investigators, first responders, and doctors with high rates of suicide…
Not only is there an acceptable market rate for trauma, it’s sometimes competitive and requires licensing.
There is one difference between first responders/doctors and the other classes (and the moderators under discussion here)
First responders/doctors/CPS investigators see the worst but they also have days where they make a difference. Save a life or multiple lives. I'm sure it's a huge part of what makes the job bearable, and to some meaningful.
I'm not discounting your point about high rates of suicide either. If anything, when you take away any good days, you're left, as a content moderator, with just seeing the worst of the world day in, day out, with nothing to make it meaningful. I'd suggest that's something we as a society should not tolerate as being an acceptable trade for the ability to share cat photos.
Emergency Department^ doctors, what do they make? give people who have to review the worst humanity has to offer and pay them that. and while we're at it, ambulance personnel should get a huge pay bump. Take it from nurses' pay.
^ i originally said "triage doctors" but i meant the resident ER doc.
OpenAI runs ChatGPT where users submit text and photos and OpenAI generates and sends text and photos back. So users could be submitting CSAM. And yes, OpenAI could be generating CSAM. It's not limited to being a pull operation. What am I missing?
What you're missing is that they're "separate" parts of the business.
The core Facebook product is users' posts. It's not possible to separate those two. Nor can one downscale Facebook in a way that stops the problem; The aforementioned "Facebook has had this problem because it's a problem we've had since the medieval days of a town bulletin board"
With OpenAI, the way ChatGPT was built and user submissions are separate things. The GPT models could have been have been trained without this mess. OpenAI could be more selective in what data it scrapes.
While OpenAI cannot stop users sending god knows what in their prompt text and images, OpenAI can choose to not interact with that data beyond the minimum legal retention, by e.g. not using it for training the next generation of models. This would massively downscale the problem.
AI output is another such problem, where A) Maybe this'd be less of a problem if they didn't recklessly include a bunch of CSAM into the training data by accident, and B) LLMs just aren't the kind of fundamental human right that "having a public opinion" is. It would be fine if they were less good, invented years later, or even not invented at all.
The main counterargument to the latter has been the "But China is inventing evil AI" spiel, which is fairly weak. If China builds an orphaned baby crushing machine, we do not need to build an orphaned baby crushing machine of our own. (And the reality is that China is only chasing AI so aggressively because the west does. They're reasonable people, it would have been entirely possible for both the west and China to make a mutual "no orphan crushing" agreement and just accept slower rollout of technology. This is exactly what has been done with human genetic engineering, and China did in fact enforce these norms.)
> In contrast, OpenAI has no such problem. It did not have CSAM pushed onto it, it actively collected such data itself. It could have, at any point before and after, simply stopped scraping all of the web indiscriminately and switched to using more curated sources of scraped data.
You've just thrown the garbage over your fence.
Instead of OpenAI contracting Sama to classify CSAM, the "Curators" have to.
At the end of the day, someone needs to classify it. If you say the platforms need to, and they miss some, and it ends up in OAI training data, OAI is going to be the entity paying the prices.
> In contrast, OpenAI has no such problem. It did not have CSAM pushed onto it, it actively collected such data itself. It could have, at any point before and after, simply stopped scraping all of the web indiscriminately and switched to using more curated sources of scraped data.
This is of course incredibly illegal, but megacorps (by valuation) and oligarchy members are above the law so who cares. I assume there could be a regulatory framework which can make this legal for an extremely specific purpose, but there is zero change that OpenAI was part of this/abiding by this in 2022, absolutely none.
CSAM exists on social media because they are so large that it's not possible to moderate them effectively. To me this is a a no-go. If a business is so large that it cannot respect laws, it needs to be shut down.
The correct way to organize social media is in federated way. Each server only holds on average a few hundred or few thousand people. Server moderators should be legally responsible for content on their server. CSAM on social media will be 100x suppressed because banning people is way easier on small servers.
Not many moderators will have to look at CSAM because the structure of the system makes is unappealing to even try sharing CSAM, knowing you will be immediately blocked.
Having tens of thousands of decentralized, independently moderated servers would result in an order of magnitude more CSAM being shared than having a few oligopolies. The abusers just have to find the weakest link, and that weakest link will have fewer resources than multi trillion dollar companies. You would also likely not hear many news stories about it, because they won't have the expertise to even detect it.
That's a tradeoff you can choose to make, but you need to enter into it with open eyes.
> Having tens of thousands of decentralized, independently moderated servers would result in an order of magnitude more CSAM being shared than having a few oligopolies.
It doesn't matter how many are shared but how many are viewed. On a small server community policing works just fine, bad actors are easier and faster to block and to top it off, the smaller reach of each server makes it unprofitable to target multiple serves, fish for their weak points. etc - the dirty jobs become unprofitable which is what matters most.
With the help of AI, small players can do a better job at removing CSAM.
>That's a tradeoff you can choose to make, but you need to enter into it with open eyes.
No it's not. It's certainly not my choice. No one asked me if it's okay for Facebook to distribute CSAM because you insist it would be worse if it didn't.
I don't really care if you classify it as a choice or not. One set of actions results in more CSAM than others. Just because you don't like the implication of there being tradeoffs doesn't mean there aren't tradeoffs.
> Server moderators should be legally responsible for content on their server.
And therefore anything that is remotely questionable will be blocked. Not just kiddie porn. Pissed off a local business with a bad review? Blocked.
Child abusers are twisted people, and I really don’t care much what happens to them, but making it impossible for them to use the internet means sterilizing the whole thing.
>And therefore anything that is remotely questionable will be blocked. Not just kiddie porn. Pissed off a local business with a bad review? Blocked.
This is already the case. There is a lot of lawful, useful, medical or educational content that is actively censured on social medias because they include words or pictures of organs while same social medias actively encourage and develop algorithm to push underage girls (and possibly boys) posting pictures of themselves in sexual poses, attires and context.
Big tech and social media networks love and push CSAM, they just hide the genitals but the content really is the same.
> a lot of lawful, useful, medical or educational content
Like what? It’s all there on Wikipedia, and for all of Wiki’s faults, I have trouble imagining what kind of useful, educational, medical information you will find on social media that is better than that.
You are just saying that physical life doesn't function. People get banned or removed from all sorts of informal and formal groups all the time because of completely illegitimate reasons. That's just human politics embedded so deeply in our psychology it will never go away. They simply move to different groups - and similarly online they can move to a different federated server.
But that's not possible in today's oligopoly of social media. An invisible algorithm will ban you, and there is no way back, and few alternates. Big Social Media is way worse from a sanitizing perspective than some federated social media.
I have no deep problem with exclusion; as you say, that’s human nature and unfixable. Making mods personally legally liable for everything that appears on their board is just insane. How many minutes are acceptable for them to see and review content? Or does everything have to be pre-approved?
I know a local blog that pre-approves every comment. He lets a lot of stuff through, because he lets people be dumbasses. If he were personally liable, the conversation would get a lot quieter.
Also, if you've gone from zero to one of the biggest coroporations in the country, and have billions to throw at the 'metaverse', I find it hard to believe that removing CSAM is where you struggle.
No. It's a legitimately difficult problem because there not all naked pictures of kids are illegal. The false positive problem is bad for business, but also generally bad even if the big social media was benevolent.
Moderators need to actually understand the context of the picture/video, which requires knowledge of culture and language of the people sharing the pictures. It's really difficult to do that without hiring moderators from every culture in the world.
But small federated servers can often align along real world human social networks, so it's easier for the server admin to understand what should be removed.
The amount of CSAM online is completely out of control. There's already nation-level and sometimes international cooperation to catch any known images with perceptual hashing (think: the opposite of cryptographic hashing) as well as other automated and manual tools.
My impression is it would take Manhattan-Project levels of effort and funds to come close to "solving" this problem, especially without someone getting on a watchlist for having a telehealth-first primary care provider insurace plan and asking for advice on their toddler's chickenpox.
Human review? Meta has small armies worth of content moderators already that tend to burn out with psychological problems and have a suicide rate where you're probably better off going to fight in a real war. (This includes workers hired by Sama in Kenya, to link back to the OP.)
I will reluctantly grant Meta that they're up against a really hard problem here.
Isn't this more about disincentivizing the posting of it in the first place by increasing the chances of getting banned? Once you have to remove it, it's too late.
>CSAM on social media will be 100x suppressed because banning people is way easier on small servers.
No it isn't. Small servers often don't have paid security or moderation, are run in anonymous fashion, and have no profit motive that can even be used to incentivize them against hosting illegal content.
That's visible when it comes to porn. There's a million bootleg porn sites on the internet hosted that show off illegal content. The only site that was ever forced to curate its content was Pornhub, because they're sufficiently large, work in a jurisdiction that has laws and can be held accountable. From a content moderation standpoint going after a million web forums is an absolute pain in the ass compared to going after Facebook.
Which is the first argument any decentralization advocate always brings up (and they're correct to do so), censorship is harder and evasion of law enforcement easier when dealing with a network of independent actors.
> Server moderators should be legally responsible for content on their server.
So if you want to send someone to jail, just talk your way into joining their server, upload some illegal content, and report them for it?
> Not many moderators will have to look at CSAM because the structure of the system makes is unappealing to even try sharing CSAM, knowing you will be immediately blocked.
Why would someone join a server with active moderation if they wanted to share CSAM with their social media friends?
They would seek out one of those servers that was set up specifically for those groups, where it was known to be a safe space.
This is what many people don't get about federated networks: The people in those little servers DGAF if you block them. They want to be surrounded by their likeminded friends away from the rules of some bigger service like Facebook or Twitter. Federated social media is the perfect platform for them because they can find someone who set up a server in some other country with their own idea of rules and join that, not be subject to the regulations of mainstream social media.
right, and you have other users on fediverse that notice that server leaking, and if the content is bad enough, report the service to an authority. Having all of the pedophiles and other creeps on a tiny subset of servers, isloated islands of them; well, that ought make enforcement easier.
It also makes it relatively easy to avoid, as server admins share blocklists. I know a dozen servers offhand that i'd block if i ran another fediverse server.
Fosstodon fediverse server doesn't have this issue, for example.
I replied this way because the way you wrote it, it sounds like an indictment of a system that's designed to avoid advertisers getting user profiles, over all else.
The problem is the people who participate in this (the illegal and immoral), and not "the network."
The one thing I will throw out here that I can add to this conversation is that I think the government simply does not care, either. It's mainly only in regard to mass public outrage, or when someone is a political target that it gets dealt with from a law enforcement level.
Anecdotally, when I was a young adult I was a volunteer moderator for a large forum. We got reports of CSAM several times a month and had a process for escalating and reporting it to the FBI IC3 - we retained a lot of information about the users that posted it.
One of the administrators of the website mentioned to me that over the years since the inception of the forum, they'd reported almost a thousand incidents of CSAM distribution - and the FBI followed up with them to get information less than 10 times in total.
The FBI is interested in busting perverts in closets. That's often how they work their way up the "supply chain" when it comes to CSAM. Consumers lead them to distributors, who lead them to producers.
Yep. If you cannot both safely and legally provide the thing you are selling you are no longer a legitimate company you are a criminal enterprise profiting off of exploitation.
Do car companies sell cars without air bags, or seat belts? What about cars that haven't been crash tested? What happens to them if they don't do this do you think?
Would you drive a car optimized for profit that didn't have those safety features? How about on a highway? Daily?
When FORD dngaf with the Pinto and Corsair( like tech companies do not gaf), they deservedly got this same level of contempt/demand for oversite. A dude named Ralph Nader went on a huge crusade about it. And they got a ton more oversite, safety requirements, etc put on them.
I am not sold on the federated thing to solve CSAM or similar issues.
Actually companies should be bullied about privacy and copyright so they are unable to share any contents at a scale with 3rd parties. Thus they have to solve it on their own and forced to realize their business model is shit.
when i ran a fediverse server for myself and 3 people, but allowed public signups if someone came by; it was very easy to ban people, and very easy to null-route entire swaths of the fediverse, because i didn't want their content on my service.
That's more what i got from that pull-quote. I know a company that has hundreds of individual forums, and those are all moderated quickly and correctly (last i heard). They're moderated so effectively they often get DDoS by Russian IPs for banning users for scam posts from that country.
These workers prepare data for AI. I don't think the need for them will go away anyway soon.
Westeners are too expensive and unwilling to do it. AI is a business model that requires poverty and extreme inequality to function. Yes other businesses do that too, but they don't claim it's a solution to everything while it actually has very special human requirements.
There are more reasons why these jobs are located in developing countries, it's not only the price of labour. Imagine for a second, these annotations would have to be done in the US. The public outrage would probably be audible across the Atlantic. This is another form of imperialism.
I agree that there’s no good way to do this other than like… no user generated content ever or just ban everyone for their baby pics and etc….and nobody can post them.
Granted the latter is kinda happening distantly on YouTube where you can’t talk about “ suicide “ so everyone self censors…
you must be extremy priviledge to think that way, even as EU I would be glad to do it for the minimum salary. For your info, a terrible job for most human is a job that is extremly hard physically at the point of destroying your health. That said, like many people, I would find it much more interesting than many boring job. [If someone read this, please hire me for this, in exchange I would work the 5 first hour for free]
Isn't it more that tech companies are just more high profile and integral to political and social landscape than older companies; but reviewing the current political zeitgeist, they're in lockstep to what some, if not all, would just call fascism?
They are literal defense and offense contractors. They hang out at the Pentagon. They sell political data to sway elections. They give gifts to leaders for favors. It is technofacism.
Safety and user pain is a part of tech which seems largely ignored, even on sites like HN.
I really have no idea why this ignorance prevails; commenters seem to genuinely be unaware of what goes on in Trust and Safety processes.
I mean, most users would complain about content moderation, but their experience would be miles ahead of what most of humanity enjoys when it comes to responsiveness.
I believe this lack of knowledge, examples, and case history is causing a blind spot in tech centric conversations when it comes to the causes of the Techlash.
Unfortunately this backlash is also the perfect cover for authoritarian government action - they come across as responsive to voters while also reigning in firms that are more responsive to American citizens and government officers than their own.
Companies of the 20th century certainly weren't more ethical. (Though a few select tech companies seem to be intent on proving the opposite.)
But it's not really a fascism thing. While fascism does love the oppression of women, and the current crop of fascists have a notable connection to the Epstein case, this is a lot more boring.
Sam Altman's not a fascist, he's a wet noodle who sucks up to the Trump administration for money. He's not even good at it. The way his company handled CSAM does cast aspersions on Altman & the accusations from his sister, but all other evidence suggests he's just a moron acting recklessly. Not identifying the problem ahead of time, and acting poorly in response.
In the case of Meta. We know who Zuckerberg is. The company got it's start as, in crude terms, a sex pest website. The original "Facemash" website forcibly taken down by Harvard. This is not some new consequence of this turn to fascism, Zuckerberg's always been like this, and the actions taken against him were clearly not enough to avoid the company culture following his precedent.
> Companies of the 20th century certainly weren't more ethical.
Disagree, not on average. There was a non-trivially higher % of decisions made based on "what's good for the customer" or "what's good for the product" or "I would be ashamed to do this" and a lower % of decisions made based on "what maximizes profit in the next quarter". I think that is more ethical. To take it to an extreme, using slave labor because it's good for the customer is more ethical than using slave labor to maximize profit in the next quarter.
Sounds about right. If you know someone who uses these smart glasses, it's important not to tolerate them whatsoever. Don't speak with them, interact with them. I wouldn't even recommend being in their presence.
I'll grant you smartphones, but smart TVs usually don't have cameras/microphones. The problem with smart glasses is that they constantly capture video and upload it to $VENDOR like in this case.
A mostly-solitary sporting event (or one where you know all the other participants and can get their consent to record beforehand) seems like a reasonable use of these sorts of glasses. I wouldn’t personally give consent just as a sort of privacy reflex, but it really depends on your social circle.
The problem is there's places where you'd get noticed and probably removed for filming with a gopro, or even a smartphone. My local "wellness center" and pools have you deposit your smartphone before you exit the changing area into the showers.
The danger with creep glasses is that many people don't know what they are, they can be used with the LED disabled so they're perfect for filming people without their knowledge, and "these are prescription glasses" has a good chance of working. In a place with a "no recording devices" policy, "could you put that gopro away" has wide social acceptance/support, "take those glasses off" less so.
There's also nothing stopping us from stigmatizing the use of smartphones in public. Even a slight discouragement of it would be progress. It doesn't have to be all or nothing.
I think smartphones are a lost cause. Even at the gym, there are guys in the locker room taking pics of themselves in the mirror. Meanwhile I'm walking ass-naked out of the shower. There is just no sensitivity to appropriate time and place anymore.
Many security cameras have the ability to record audio. Depending on where you are, it might be illegal to use it. All the cams I have purchased have it. That would include ReoLink and a recommended model from the Frigate site.
Because person wearing glasses usually can move and video surveillance cameras usually can't?
If that's not it then spell it out for me, please.
Also, why would i be deceptive in this discussion? I feel like I missed some ideological conflict.
Imagine someone pulling up a smartphone and then recording everything that happens around them. Contrast that with someone wearing smart glasses and doing that exact same thing.
On a separate note, (and this is a genuine question) are you by any chance aware the term Non-consensual intimate imagery / NCII?
I am beginning to suspect that the average HN goer isn’t aware of the scope and scale of the Trust and Safety problem.
They don't care. Or they refuse to realize that tech isn't the solution to it, but an amplifier of it's scale.
Can tell you that my urge to take photos/record drastically dips around other people. Particularly if it were meant for any sort of commercial exploitation. Stephenson called people wired for max indiscriminate data collection/processing "gargoyles". Personally I prefer glassholes.
Someone pulling up a smartphone on me would feel hostile because it's violating a social contract. Maybe I'd feel betrayed and attacked if it turned out someone was recording me using glasses, but I don't know, I don't care about dashcams and this is not that much different. I imagine it feels bad and scary for women when someone takes creepshots of them, and this tech does open opportunities for that. Maybe that would be enough for me to hate glasshats if I had a bit of empathy. But isn't the genie already out of the bottle with 'deep nude' models available for everyone forever?
No, i don't think I've heard about NCII before, and Trust and Safety sounds like some corporate PR whitewashing term to me.
Most people don't run around holding out their smartphone directly in front of them. It has to be pointed at the subject, and tends to be obvious.
Smart glasses, however, are always aimed at whatever the wearer is looking at. They may or may not be recording (note the reports of people hiding the LED indicators), and at a fair distance could easily be mistaken for a normal pair.
The general populace is much more likely to notice the former recording rather than the latter.
I've seen people keep their phone in their shirt pocket. The only reason it tends to be obvious is that most people aren't trying to be covert. Those aren't the ones you should be worried about.
Don’t forget that audio recording is a thing. The camera doesn’t have to be pointed at you to violate your privacy. Plus I bet you walk past 90% (or more) of all cameras without ever noticing them. You only notice someone’s glasses because they are novel, not because they are more likely to record you.
At everything on the opposite side of the screen, typically. There is a recording light for Meta glasses, but not one for iPhones, for example: the "recording" indicators are all user-side there.
A Kenyan workers' organisation alleges Meta's decision was caused by the staff speaking out.
Meta says it's because Sama did not meet its standards, a criticism Sama rejects ...
Well, yeah. If I went straight to the press to trash the reputation of my client's product, rather than communicating internally first to help them proactively address the issues, I would expect to get fired.
Not that I am remotely interested in defending Meta, or optimistic that they would proactively address privacy issues. But I don't feel that sympathetic to the outsourcing company here either.
I don't know what happened behind the scenes. I'm just going off what is said and not said in the article. If I were whistleblowing about something like this, I would take pains to describe what measures I took internally before going public. I didn't see any of that here.
EDIT: Look, to be clear, I think it's bad that naive or uninformed people are buying video recorders from Meta and unintentionally having their private lives intruded on by a company that, based on its history, clearly can't be trusted to be a helpful, transparent partner to customers on privacy. I think it's good that the media is giving people a reminder of this. I think it's good that the sources said something, even though the consequences they suffered seem inevitable. But to me, there is nothing essentially new to be learned here, and I don't know what can or should be done to improve the situation. I think for now, the best thing for people to do is not buy Meta hardware if they have any desire for privacy. Maybe there are laws that could help, but what should be in the laws exactly? It's not obvious to me what would work. I suspect that some of the reason people buy these products is for data capture, and that will sometimes lead to sensitive stuff being recorded. What should the rules be around this and who should decide? Personally I don't know.
What makes you think the outsourcing firm didn't raise these concerns in email or meetings? You think these people wanted to lose jobs and income? That's irrational.
Why reflexively defend a massive tech corporation caught repeatedly violating the law?
There are transgressions severe enough that your duty to stop them is heavier than your responsibility to "the reputation of your client's product." Amazing this needs to be stated, frankly.
More like a bright future being someone's fall guy. The ignorance to think that a large tech giant like Facebook would give a crap about any of those concerns makes this person too politically inept to make it anywhere
What specifically do you mean? It is by design that smart glasses see the things happening in front of their users? Yes, it is. That is why people buy them.
Huh. There you go again, thinking everyone else is an idiot. Capture of video data of users by Meta is never acceptable. It would not be acceptable for any phone, and it is not acceptable for any glass, ever.
Saving the data for any purpose other than allowing users to access it is bad enough; allowing Meta employees or contractors to view personal videos is on a whole new level.
I don't know why people buy smart glasses. Maybe they buy them for video capture. If so, the videos go to Meta's servers and Meta might do things with them. They might be criticized for not reviewing them in certain cases. That's one reason why I wouldn't buy Meta smart glasses.
The main issue here is Facebook employees viewing users' private video streams (including of user nudity) without the users' knowledge.
The secondary issue is that it's generally frowned upon to make your employees view nudity in the workplace. Are there extenuating circumstances here? No, we have no evidence there are any extenuating circumstances here.
OpenAI had them classify CSAM, so Sama fired them as a client back in 2022. https://time.com/6247678/openai-chatgpt-kenya-workers/
We're 4 years on, 3 years since that report broke. Not a single thing has improved about how tech companies operate.