[diamondhandle]

People who comment on these moderator issues always overlook the huge huge mental red flag that these moderators always discuss: images of children being molested and tortured.

Endless images of young children. Being sexually assaulted, penetrated, made to do degrading things.

Think about how that statement makes you feel, and then imagine you’ve been tricked into looking at this stuff for money as a “content moderator.” It’s a form of cruel economic punishment.

Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. They are the ones who should be forced to pay for the long term mental health of these workers.

California ballot initiative, anyone?

[stickfigure]

You know who else has to deal with these images - and much worse, real-world living cases? Law enforcement. Lawyers. Juries.

Do we just sit back and say "let's stop subjecting those poor officers from having to see evil things"? Of course not. We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life.

See also: Soldiers. Firefighters. Emergency medical workers.

Personally, I'd rather moderate gross things than, say, work in a coal mine. But that's me. I don't appreciate the insinuation that this makes me some sort of monster.

[q3k]

> We hire people with strong stomachs and a strong sense of duty.

It's difficult to have a strong sense of duty for the megacorp that doesn't even bother to employ you directly, but pays another megacorp which then pays you peanuts. When you don't even get to have the context of what you're doing, don't get to follow up on whether what you did was meaningful, whether the perp was caught and tried, whether the child was saved.

It's difficult to have a strong stomach if you don't see this as an occiasional element of your generally tame and meaningful work, but see hours upon hours, days upon days of some of the worst shit that the human race is capable of.

[stephenhuey]

Good point. If someone joins the Marines, they are not paid much but I assume they have a sense of pride in being a Marine. Contractors working for the military are not officially employed and often they're put to work on projects we do not want our military employees doing, but they do it for much higher pay. Law enforcement officials generally have a similar level of pride in their service and are recognized with a badge, uniform, respect, etc. If the contractors that Facebook is putting to work on content moderation don't really get to be part of the Facebook employee roster but are having to do difficult work on par with what many law enforcement personnel have to handle, should they perhaps be compensated far more than Facebook employees just like how a black ops contractor is paid far more than a uniformed Marine? If that's too expensive for Facebook, then perhaps Facebook is not running a profitable business and we should just let it fail.

[kodah]

> If someone joins the Marines, they are not paid much but I assume they have a sense of pride in being a Marine. Contractors working for the military are not officially employed and often they're put to work on projects we do not want our military employees doing, but they do it for much higher pay.

I know this isn't what you're doing, but this sentiment does come awful close to, "they know what they signed up for" logic. Most Marines know before they sign up that they are signing up for a high risk job, mentally and physically. In bootcamp you go through fairly extreme (to an every day person) conditioning and you are regularly conditioned to stress throughout your tenure. It takes an insane amount of work to make a person disregard concern for their own safety and run towards the sound of gun fire. Even then, this conditioning is problematic when people get out, due to numerous reasons, which is why (imo) you see so much disarray in veteran communities. Conditioning people to be to handle extraordinary circumstances is necessary, imo, but you have to find a way to decondition them.

[ceejayoz]

"We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life."

https://www.theatlantic.com/national/archive/2014/09/police-...

> Two studies have found that at least 40 percent of police officer families experience domestic violence, in contrast to 10 percent of families in the general population. A third study of older and more experienced officers found a rate of 24 percent, indicating that domestic violence is two to four times more common among police families than American families in general.

[baumy]

Those studies have been thoroughly discredited. Not only do they qualify things like "shouting" and "loss of temper" as domestic violence (which of course do not meet the legal threshold for that term), they have never been able to be reproduced. When the same exact researchers attempted to reproduce, they got a wildly different rate of 24% using the exact same methodology. Other researchers using better methodology and appropriate definitions of "domestic violence" have found numbers generally between 7% and 11%.

The 40% study itself even says that the victims reported a 10% rate of physical domestic violence from their partner, but specifically does not indicate who the aggressor is - the officer or their spouse. The study is also 30+ years old at this point. [0]

More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime" [1]

Other studies [2] [3] report rates between about 2% and 11% for mixtures of one time and recurring domestic violence, which are in line with population averages.

[0] https://babel.hathitrust.org/cgi/pt?id=umn.31951003089863c

[1] http://scholarcommons.usf.edu/etd/1862

[2] http://webapp1.dlib.indiana.edu/virtual_disk_library/index.c...

[3] https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...

[ceejayoz]

> More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime" [1]

I'm sorry, are we reading the same thing?

"Using a survey created from existing scales, 250 officers were contacted within several departments in Central Florida, of these, 90 officers responded."

Asking cops "do you do crimes" is an interesting methodology, for sure.

[baumy]

> Asking cops "do you do crimes" is an interesting methodology, for sure.

How do you think the population wide numbers are gathered?

The core point is that the 40% number is exaggerated BS, if for no other reason than it literally counts "shouting" and "loss of temper". There's no reliable evidence indicating the number for police officers is any different than the general population.

[ceejayoz]

> How do you think the population wide numbers are gathered?

Not merely by asking the perpetrators.

13% of active cops admitting in a survey to domestic violence certainly means the actual number is higher than that.

[nieve]

> More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime"

Which means 12-13% _admitted_ committing physical domestic violence, what has to be a significant undercount.

[FireBeyond]

> See also: Soldiers. Firefighters. Emergency medical workers.

As a firefighter and paramedic, I wouldn't even begin to compare my job, even dealing with the occasional horrible things, in the real world, with a job that consisted of hours on end of viewing things that would have been deleted from rotten.com for being too problematic, and the effects that that would have on my mental health.

[intended]

I mean, content moderation would be comparable after it had 500 years to evolve human centric best practices.

Even police forces have developed unions and employee care and training practices to keep them sane and healthy.

In contrast, for CM you have the magic of the market outsourcing the work to replaceable humans.

To be honest this assertion of “strong stomachs” is a unsubstantiated and unhealthy fantasy of how people in stressful jobs handle that stress.

[nickthemagicman]

What 'best practices' developed over 500 years are you referring?

I have numerous friends who are police.

They had three months training, then were sent out to see the worst humanity has to offer.

One of my police friends has a coffee table book of interesting dead bodies from crime scenes.

I'm not sure what 500 years of best practice you're referring.

[intended]

From crime scene handling, to forensics, to specialists like SWAT, all the way to non gun carrying police (uk/colonies) vs America’s gun carrying cops to access to medical and mental care and even labor unions, these are modern inventions.

Compare to things like Spanish brotherhoods or the military handling policing in Rome.

[nickthemagicman]

It's a stretch

[cgriswald]

Even someone on a child porn task force won’t be looking at these images in equivalent quantity, frequency, or duration of a content moderator. Many of the things this former content moderator is suggesting are already available to the groups you mention and many people, myself included, don’t think we take good enough care of those groups. It’s also important to mention that stopping child porn is a more important goal by any sane metric than removing child porn from a website so the website can make money tracking people for advertising dollars.

[AdrianB1]

You need to have the stomach to do some jobs. I remember the time when many of my friends in the medical school asked me to go with them to the dissection lab in the first year of college; I was doing CS at that time, but the professors did not know the students yet, so they gave me a lab coat and went with the group. 10 minutes later, a few students fainted, some were puking in the corners, the professor asked me to put the liver in the body on the table because I looked the least affected. I did it, so he asked me for the name to give one extra point at the exam for the semester (I gave a friend's name that was puking).

Most of these people grew up in cities, I grew up in the country side cutting chicken or pigs for food; this makes you less sensitive to some extent.

My friends that are now MDs told me that year was the worst experience of their lives. Being a Facebook moderator can be worse than that, so it can be really bad for some of the people doing this job.

[type0]

> My friends that are now MDs told me that year was the worst experience of their lives.

This is just incredibly telling how shielded these people are.

[at_a_remove]

I think that selecting the "right" people may be key here.

Some time back, the child of some friends died, rather badly. This was a child I knew. They asked me to go through the police report because they could not bear to do so, just to see if anything was missed. In the interests of shielding them, I did. I had to detach in a peculiar way, and that may be while I have been able to attend autopsies and such.

Could I do it forty hours a week? I don't know, I won't claim to it, but I think looking for some kind of quality like that would be where to go until AI takes this over -- and frankly I think that will be a long time coming.

[jka]

> Not everyone who sees this stuff is scarred for life.

Psychological trauama and PTSD appear fairly prevalent for first-responders (worldwide)[1].

And first-responders tend to receive a lot of community support, respect and hazard pay in return for what they do.

[1] - https://en.wikipedia.org/wiki/Trauma_and_first_responders

[mschuster91]

> Do we just sit back and say "let's stop subjecting those poor officers from having to see evil things"? Of course not. We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life.

I happen to know cops. The suicide and dropout rate for those in IT evidence and sexual crimes departments is enormous.

[lillecarl]

I don't understand why on earth anyone would use Facebook as a platform for sharing this kind of content, they know It'll be removed and that Facebook is quite good at tracking you, so unless you're using a VM connected to a VPN with a browser soley for this content you'll have the police called on you?

It just doesn't make sense to me. Also, why is it that facebooks content moderators have to look at everything while Google just removed content and says nothing more about it? Does Google have more confidence in their AI while Facebook doesn't and therefore need people to manually check it? Or is it that there might be more content on Facebook uploaded that an AI would misinterpret as child abuse (baby photos and the likes that people seem so desperate to share on Facebook these days).

[rscoots]

This podcast from last year answers all these questions. Although the answers are not encouraging...

Very eye-opening stuff: https://youtu.be/qv_hokG2oSo

[fighterpilot]

They use a number of platforms for this but FB is quite aggressive about cracking down on it, unlike AWS among others who mostly turn a blind eye.

[commandlinefan]

I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health. I'm sure numbers are hard to come by, but I'd have guessed that there wouldn't be more than a couple of these cases per month - the problem can't be that widespread, or you'd assume it would spill over into other areas as well.

[diamondhandle]

It’s way, way more common than you’d ever want to believe.

Here’s a starting point, from one very repressed culture where the act of going to the police puts great shame on your family and turns you into an outcast:

> There seems to be no sign of decline in the number of sexual abuse cases against children. According to the Ministry of Health, Labor and Welfare, child consultation centers nationwide responded to 2,077 cases of child sexual abuse in fiscal 2019, up 2.75 times from fiscal 2000, when the Act on the Prevention, etc. of Child Abuse went into force. In 1,056 of these cases, the perpetrators were biological fathers.

https://mainichi.jp/english/articles/20210322/p2a/00m/0na/01...

Please, listen to children and believe their stories.

[pdpi]

Low frequency outcomes over large number of events can produce surprising numbers. 0.01% of 2B people is still 200k people. If each of them posts a single image once a year you’re still going to get around 600 pictures of child abuse per day.

[mbg721]

There doesn't have to be that much of it created, if it can be copied for free, and someone is willing to automate posting it. Cynical-me totally believes there are enough people doing that to traumatize a human content-sorter.

[lillecarl]

YouTubes contentID system is quite good at detecting just a couple seconds of sound to determine if something is copyrighted or not. It should be quite easy to match copies of content and just get rid of it straight away, meaning they don't have to see duplicates. I would assume they already do this. Which leads me to believe there's actually that much content being produced, I imagine most of it is not produced in the West though.

[larrywright]

There is a national database of child porn content that most big companies that host video and photos use (Facebook, Twitter, YouTube, et al). It allows them to filter the known content. I don’t know how fast that gets updated though. The larger issue is the new content. Facebook makes it very easy to take and share video, and makes producing and sharing this content easier.

[wiedelphine]

It feels like you are suggesting that under a certain amount looking at images of children being sexually abused or tortured shouldn't have an impact on mental health? For me personally I think even a single image like that would haunt me.

[pdpi]

Seeing one image once will have an impact but probably allow you to still lead a normal life after that. Spending all day looking at that sort of thing over an extended period of time is much more likely to have serious long-lasting effects.

I don’t think there’s a magical line where it suddenly has an impact on mental health, and the details will vary from person to person, but there is most certainly a point where you go from functioning adult to justifying a formal diagnosis of PTSD or something similar.

[sethammons]

I don’t know about you, but I’ve accidentally came across a few pics on an old hard drive I was given second hand. I couldn’t even use my computer for a couple of weeks and I can still feel the revulsion decades later.

[sethammons]

I'll burn some karma here: I know one is not supposed to question down votes, but here it goes. Why would someone down vote my comment? If someone doesn't think it adds to the narrative, skip the vote. If you feel it actively detracts or presents a false narrative, down vote.

> I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health

And I reply that just a few images had an outsized impact on me. And down voted. Normally, anecdotal evidence is replied to with the general response that "the plural of anecdote is not data."

Maybe the down note is that I'm not referencing mental health? Am I supposed to connect all the dots in my reply and go from "if a few images can have a visceral impact on me, that it is within reason that much more exposure especially over time could lead to mental health issues"?

[Anthony-G]

It seems like some of your sibling comments have similarly been down-voted. God knows why. I’ve upvoted yours and theirs because I consider personal anecdotes to be useful contributions to discussion. If discussions were to be based solely on peer-reviewed meta-analyses of empirical data, they would be poorer for missing the colour and detail – and far, far drier.

[timthorn]

Here's a currently open job role for a content assessor at the Internet Watch Foundation: https://www.iwf.org.uk/what-we-do/who-we-are/work-us

The stats there say they've dealt with 1.2m reports over 23 years, with 560,000 takedowns. That's certainly more than a couple a month.

[samstave]

Are you kidding me???

When I was at FB ~2012+ -- I commuted in with one of the content mods on the CP/abuse -- and shit I heard from him was HORRIFIC. and there is TONS of it.

And with all the pedo-network shit coming to light these days.. child abuse is WAY WAY WAY worse than anyone can imagine on this planet.

Ill take your downvotes without comment as a tacit acquiescence to child abuse.

[commandlinefan]

That's surprising to me. I believe you, but I'm surprised. I would have assumed that this information would have been turned over to the police and the perpetrators whittled out of the general population in short order.

[samstave]

They do turn stuff over, I have no idea of what % is though...

They should be transparent about it, IMO - as we really need a growing outcry on child safety rights.

We constantly talk about the rights and protections for all these various groups, yet, in really really dire situations like this, sex trafficking etc, by comparison is way more important than kvetching over identity pronouns and such.

We need a global anti-pedo-abuse force.

[diamondhandle]

I am really glad you posted this, as I’ve heard similar from people within the org. It becomes hard to justify working on such products when you know you’re perpetuating so much harm, with zero interest from leadership in resolving the issues.

You might be getting downvoted by your former colleagues. Would you be surprised?

[nitwit005]

> Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible.

Even small sites have people upload child porn and other stuff into them, and some IT guy has to go handle it.

The only thing that can really reduce the scale of this stuff is for the people doing it to suffer a consequence like jail time. That's clearly not happening in most cases, despite Facebook and others sharing information with the authorities.

The same is true for other issues like people being hounded by death/rape threats and the like online. The police mostly ignore it as an issue, so moderators have to read a bunch of it, and the people doing it just get a suspension or ban from the one platform.

[chrisjc]

> They are the ones who should be forced to pay for the long term mental health of these workers.

I don't disagree... However, this doesn't do anything to address the issue that someone still has to sift through this disturbing content, and society as a whole probably still pays the price.

I don't think any sort of compensation or mental help would prevent the kinds of side effects this sort of role is exposed to.

[mc32]

I don’t know why FB believes it should aim to be the repository for all of everyone’s media.

Most photos and videos are crappy. They could throttle it to 10 per person per day and increase quality. I’m sure people would complain and FB would think they lose out to a competitor with unlimited uploads. However, a reduced load means easier job to address the bad stuff.

It’s probably unworkable, but making people more discerning rather than less isn’t a bad thing.

[nonbirithm]

> Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible.

Is there any venture capitalist that would generally believe "there is value in not trying to connect everyone on Earth together?"

It feels like when you market your hot startup idea, connecting the world or enacting change is merely a cool new thing that carries no long-term downside. Find a market, connect with a target audience, grow. Maybe that's how Facebook started (or not).

I feel that, because of human nature, if it weren't Facebook then some other company would have inevitably taken its place. Being able to connect to someone faster and easier than ever sounds like a tangible gain for average people. The idea, with no strings attached, was too tempting to pass up for someone that truly wants to change the world and be a hero of some kind. It worked for a single university, after all. It's the sense of starting a technology company when the future is completely unknown and the potential for change is exciting. I'm not sure there was anyone that would have understood the consequences of taking such an idea to its logical conclusion, and now Facebook has become irreversibly baked into human culture.

The more people that want to connect, the more moderation staff is needed to match. Unfortunately, the desire to connect with others is wired into pretty much everyone on Earth at the deepest level (with exceptions, of course), and so your potential market is... pretty much everyone on Earth.

[coryrc]

Why not hire the sickos who like these things to do the filtering?