[mortenjorck]

This is a very hard problem that YouTube and Facebook made for themselves by becoming the world’s largest advertising platforms. They depend on engagement for ad revenue, they designed world-class algorithms to promote this engagement, and it turned out that extremist content happens to be very engaging.

And so the problem is, building an algorithm that blindly promotes whatever keeps users on the site, for all its complexity, is a far more tractable problem compared to building a system that can avoid promoting content that promotes violence. In the meantime, they throw armies of people at the problem, to moderate content and respond to user reports, but it’s a losing battle.

They had the technology to create a monster, but don’t have the technology to stop it.

[skybrian]

Except that advertisers are not interested in sponsoring extremist content, so you could argue the incentives go in the other direction.

[kragen]

YouTube permits a lot of demonetized videos they don't dare to sell advertising on, or (previously) that YouTubers wouldn't give permission to sell advertising on, because the alternative of erasing the videos would send the YouTubers and their viewers to DailyMotion or Vimeo. As long as the demonetized viewers occasionally watch a non-demonetized video, or the sometimes-demonetized YouTubers occasionally make non-demonetized videos, YouTube makes more money that way.

[skybrian]

That's a weaker incentive, though. And it comes with a risk of occasional scandals that scare away advertisers, at least temporarily.

I don't think incentives are enough to predict what YouTube will do. The future is uncertain, there are a lot of policies that could plausibly make money, and the decision-makers are people who read the news and have opinions about what's right, not paperclip maximizers. Politics matters as well as economics.

[kragen]

True!

[bsder]

> They had the technology to create a monster, but don’t have the technology to stop it.

They don't have the incentive to stop it.

All you have to do is make posting have an actual price, no matter how nominal, and most of this goes away.

Unfortunately for their profits, it will wipe out 90+% of their user base. And it will kill virality cold (you won't forward something to your whole address book if it costs you a dollar).

[ocdtrekkie]

The other incentive option is to charge the platform for the violation. Aka, remove Section 230. If the platform was to be charged for posting dangerous content, the platform would magically be able to largely prevent it overnight.

[qball]

>Aka, remove Section 230. If the platform was to be charged for posting dangerous content, the platform would magically be able to largely prevent it overnight.

That's not actually what Section 230 does. It prevents the platform from being charged if while moderating it makes a mistake.

Removing Section 230 would do more harm than good if you consider censorship good- because then the only option to escape legal liability would be to allow everything. Remember that Section 230 was part of the Communications Decency Act and was designed to ensure that websites could remove pornographic and objectionable content without being responsible (of course, it's also the only part of that law that was found constitutional, and for good reason).

By taking the only legal immunity for half-baked censorship away (i.e. private company gets to moderate what's posted but won't suffer consequences for getting it wrong) you've closed the door even harder on it- US law won't actually let you censor consequence-free otherwise, which is the whole reason S.230 was added in the first place.

[wbl]

Charge them with what? In the US there is nothing illegal about being a Nazi. There is nothing illegal in selling Mein Kampf etc.

[gcb0]

please don't compare selling a book with being a nazi.

[chris_wot]

Nor compare reading a particular book makes one a Nazi.

[gcb0]

hum.. that's exactly the point of not comparing both things.

Reading a book, as bad as it is, start a well informed debate, that might improve the world. You can't critique or remedy something you doesn't know exists.

While hating a particular group of people (because you read the book, or didn't pay attention in life, or any other reason) does not improve anything.

[kragen]

Suppose you have two video-sharing platforms which use basically the same recommendation algorithms (etc.) to keep people engaged and clicking on videos, and they're equally effective — except that one of them censors recommendations based on some political criteria, for example, being "extremist", such as whoever the modern equivalent of Martin Luther King is. Let's call the censored platform "JedgarTube" and the other one "MLKTube", for lack of better terms. There are a couple of possibilities:

1. Those recommendations were actually less effective at keeping people engaged than whatever recommendations replace them on JedgarTube. In that case, MLKTube needs to copy the decision in order not to lose users gradually to JedgarTube. In fact, whichever platform blocks those recommendations first will experience improved user growth and engagement. Essentially the recommendations were just a bug of a primitive recommendation algorithm.

2. Those recommendations were actually more effective at keeping people engaged than whatever recommendations replace them on JedgarTube. In that case, JedgarTube will gradually lose users to MLKTube, again, assuming the platforms are otherwise equal.

Of course, the truth is that any particular censorship decision could fall into either #1 or #2. The #1 censorship decisions will be copied by MLKTube, if they aren't too incompetent or principled, while the #2 decisions will gradually accumulate into a competitive disadvantage for eyeballs at JedgarTube.

That's why the media platforms and government censors are trying to set up a global censorship system — whichever platform steps up first to be JedgarTube will lose viewers to whoever's censorship implementation is a step or two behind.

Note that none of this logic depends on normative judgments such as "extremist content is bad", "extremist content is good", "people should have freedom of speech", "people shouldn't have freedom of speech", "platforms shouldn't manipulate people with algorithmic recommendations", "algorithmic recommendations are good for people", or anything like that. It's purely reasoning about objective causes and effects about people's behavior, although I've chosen my terms to weaken the evident bias against "extremism" in this discussion so readers can reason about these causes and effects instead of being thrown around by their emotional biases.

[stcredzero]

That's why the media platforms and government censors are trying to set up a global censorship system — whichever platform steps up first to be JedgarTube will lose viewers to whoever's censorship implementation is a step or two behind.

The market isn't going to do what we want, so we're going to implement it using centralized power instead.

I didn't elect these big companies to censor the public discourse, and it's highly disturbing that they're working hand in hand with the government to do this.

Note that none of this logic depends on normative judgments

The "logic" depends on the normative judgement that what would result from a free market would be bad. There have been too few big players doing too much meddling and manipulation of the markets contained within their walled gardens to know whether that would be the case or not.

[kragen]

I didn't elect these big companies to censor the public discourse

Well, there are things you can do about it, but I didn't want to get into that when describing the incentives that make it difficult or impossible for them to censor the public discourse unilaterally, but possible to do as a cartel. I think it's important for people to have a solid factual understanding of what's going on in order for their normative judgments, and the plans they make based on those judgments, to be well-founded.

The "logic" depends on the normative judgement that what would result from a free market would be bad

No, none of the cause-and-effect relationships I described depend on anything being bad. They function in exactly the same way regardless of whether that would be good or bad.

[stcredzero]

Well, there are things you can do about it, but I didn't want to get into that when describing the incentives that make it difficult or impossible for them to censor the public discourse unilaterally, but possible to do as a cartel.

If you are acknowledging that a cartel in collusion with the government is acting to censor everyone's speech, that's at least a start.

No, none of the cause-and-effect relationships I described depend on anything being bad. They function in exactly the same way regardless of whether that would be good or bad.

The sneaky conceit is that you've constructed a causal scenario that presupposes the undesirability of a supposedly inevitably caused future. If we had more commerce, more free speech, more sharing of culture, and more cultural change, we would have less extremism. History shows us this quite clearly. It's when big, centralized powers start mucking about with the lives of individuals, that extremism rears its ugly head and becomes a problem.

[kragen]

What leads you to believe that I presuppose the undesirability of people supposedly inevitably having access to Martin Luther King? Perhaps you aren't from the US, so you don't understand the cultural context of my example of extremism? Or did you mean that I am claiming that the formation of a censorship cartel is supposedly inevitable? I wasn't claiming that; I was claiming that in the absence of a censorship cartel, people will tend to move to platforms whose recommendations aren't censored, and so they will have access to Martin Luther King and other extremist content.

[stcredzero]

I presuppose the undesirability of people supposedly inevitably having access to Martin Luther King?

I'm tiring of your willful redirection of referents, which I think is the point of your construction. Your "logic" (that basically amounts to censorship is necessary because it's inevitable that bad people will win) smacks of the same mental gymnastics people used to "prove" the existence of god.

people will tend to move to platforms whose recommendations aren't censored

Yes, and this will eventually result in MLK (whose positions are now centrist) winning and authoritarians on both extremes losing. Your scheme uses a lot of words to attempt a pretty transparent sleight of hand.

[rcoveson]

> They had the technology to create a monster, but don’t have the technology to stop it.

At first I was going to argue that they could stop it and that they did have the "technology" to do so, but the more I thought about it the more I concluded that in the system we have built, they really cannot. Profitable ideas are unstoppable until they are proved less profitable than some other idea, or regulated out of existence. Fiduciary duty enshrines this into law. Sufficient competition ensures that it is the only winning strategy.

Obligatory SSC: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

[deogeo]

> Fiduciary duty enshrines this into law.

A common misconception: https://www.nytimes.com/roomfordebate/2015/04/16/what-are-co...

[astrange]

Facebook doesn't have fiduciary duties to anyone, unless they're registered financial advisors.