[CM30]

Moderation is one of those things that basically cannot scale (properly) without human level AI or an absolute TON of resources. Any generalised system lacks too much context to do a good job, and any specialised one is too expensive/resource heavy. You can somewhat get away with it on something like Reddit where communities moderate themselves, but that brings its own problems for both the platform and the communities on it.

[kmeisthax]

Moderation absolutely can scale, platforms just don't want to pay for it. For two reasons:

- Moderation is a 'cost center', which is MBAspeak for "thing that doesn't provide immediate returns disproportional to investment". For context, so is engineering (us). So instead of paying a reasonable amount to hire moderators, Facebook and other platforms will spend as little as possible and barely do anything. This mentality tends to be enforced early on in the growth phase where users are being added way faster than you can afford to add moderators, but remains even after sustainable revenues have been discovered and you have plenty of money to hire people with.

- Certain types of against-the-rules posts provide a benefit to the platform hosting them. Copyright infringement is an obvious example, but that has liability associated to it, so platforms will at least pretend to care. More subtle would be things like outrage bait and political misinformation. You can hook people for life with that shit. Why would you pay money to hire people to punish your best posters?

That last one dovetails with certain calls for "free speech" online. The thing is, while all the content people want removed is harmful to users, some of it is actually beneficial to the platform. Any institutional support for freedom of speech by social media companies is motivated not by a high-minded support for liberal values, but by the fact that it's an excuse to cut moderation budgets and publish more lurid garbage.

[Sohcahtoa82]

I think you grossly underestimate how much bad actors can abuse a system and how many bad actors there are.

When attacks can be automated, but moderation can't be, it is impossible to win against bad actors no matter how much you fund the moderation team.

[throwaway345353]

Moderation can't scale over populations around the world with entirely different attitudes and behaviors. Enforcing a global moderation consensus is about as effective as the UN.

[CM30]

The problem is also that standards for what's 'acceptable' discourse vary wildly between countries and regions and subcultures and other groups. This might theoretically be ignorable by saying "well, everything will be run based on what's acceptable among tech bros in Silicon Valley", but that might in turn annoy many governments in Europe and Australia and South America and Asia for either being too lax on the restrictions/moderation or too strict on it.

For example, there are certain words which are highly offensive in some regions, but completely fine in others. Some pictures and imagery which are banned or seen as offensive in some places but not others, and other such differences.

There's also often a lot of subculture specific terminology that might come across as offensive to those outside said culture if they don't know the context or situation too. We've seen that with certain tech related terms in the past.

Political misinformation and outrage bait is similarly not always simple to classify. Is content from a satire site that too many people take as real 'misleading'? I know there have been problems with things like The Onion, the Babylon Bee, Hard Drive, etc where people took their stories at face value. It also often falls into 'things the government/those in power don't want others questioning' too, especially when it comes to certain political events and controversies.

Let's not even get into talking about conflicts of interest and scenarios where business value and fairness clash, like when certain political figures say things that would probably get any other account suspended. What if the US president, UK prime minister, president of the EU or some country leader promotes information that's abusive, misleading or dangerous? Twitter had to deal with that problem back in 2016, and just ended up pretending he didn't exist until his term in office was over.

And then there's just context. Is an insulting message aimed at someone an attack by a rival or bully or troll? Or banter from one of their friends, like how many friend groups and families can make friendly jokes at each other's expense? On smaller communities this isn't much of a problem since the people there know each well enough to tell the difference between personal attacks and jokes, but is that going to be the case with a moderator here?

So moderation gets kinda tricky due to all the context needed to know whether a message is abusive or not. A well moderated large platform probably needs people in a variety of locations, from a variety of backgrounds, with some sort of way of getting a group consensus if any staff member is unsure.

Of course the other incentives you point out don't help much either. Google and co want to automate everything, so the idea of using humans to tell the difference between quality and spam/abuse is never even considered by them. Some things like outrage bait are definitely supported by the platforms for their addictiveness, and then we get situations where the owners themselves are horribly biased/trolls/whatever and are happy to allow abuse so long as its from 'their team'.