[ethbro]

Just because there are false positives or one company has a bad implementation doesn't make it an unsolveable problem.

As parent mentioned, ml sentiment analysis + logging + peer reporting + final human analysis = problem effectively combated

It's the final human analysis that companies are loathe to fund (e.g. Facebook), as costs scale with user count.

But it's not efficiently unsolveable with current tech. Therefore, companies simply aren't prioritizing it. And won't, as long as the impact of toxic users isn't impacting the bottom line.

[shawn]

Kay. What are the odds that some HN commenters are going to solve the same problem that has been an active priority for years of the people in the field?

I can’t speak for Valve or Riot, but at S2 it was a concerning issue. There’s just no good way to do it when the people involved are actively malicious. If you think there is, get ready to have your community collapse around you as everyone complains about unfair bans.

I don’t think you really appreciate the scale of the problem. Final human analysis is not possible when there are literally millions of games per week. It’s also not something that ML can identify cleanly — the moment it does, the culture will adapt to bypass the evaluator. It always does.

[ethbro]

I absolutely appreciate the scale of the problem, and the adversarial nature of peer reporting. My day job has those same characteristics.

Millions of games per week * 30 minutes per game * avg lines of chat per minute = manageable w/ a proper streaming architecture

Especially when you have access to a massive, perfectly-scaled, distributed edge compute system. (i.e. running minimal, performance-optimized models on users' opponents' clients to do the initial detection / filter / compression pass)

But my point is this is fundamentally an economic problem, given current state of the art, not a technical one.

Companies are looking for pure-technical solutions because they're cheaper, and then complaining that it's a hard problem because they're unwilling to properly fund hybrid systems until state of the art can deliver.

ML is a first order approximation of human ability, not a magic unicorn that gives you exactly what you want. Thats the definitions of engineering: how do I build a system that fulfills my requirements from the pieces I have, not the pieces I wish I had?

So I don't feel much pity when companies allow toxic user bases to flourish because it's cheaper than funding solutions.

* Above intended in no way to belittle the awesome work folks are doing in the space with ML detection. But sometimes as engineers we need to admit when management is making unethical choices for financial gain

[shadefinale]

On the base level, it is not very hard to examine a chat log and glean who is being blatantly toxic from the log.

A big problem with games like League of Legends or Dota 2 is that you can easily be toxic or cause your teammates to be toxic without chatting or using voice comms.

There are very common trolling methods that do not require any use of chat with the express purpose of trying to incite toxicity in other players, some blatant, and some not.

However, the bigger problem is that honest mistakes can be misinterpreted by your teammates as toxicity:

Losing a close 1v1 vs your laning opponent

vs

Accidentally going too deep into enemy territory and dying once.

vs

Getting killed while attempting to secure map control for your team.

Vs

Playing too aggressively and overextending and dying many times over the course of a game.

When things like this happen, your own teammates may become upset at your poor performance and begin to lash out.

The biggest problem here is that in these games, it can feel like you have no agency over the outcome of the game when your teammates do not perform at the perceived skill level you have of them.

This is where toxic players become hard to deal with. They will start doing things that will incite toxicity in their teammates while maintaining plauisble deniability:

- Confusing teammates by providing useless or inaccurate information about the current gamestate. (Pinging, map calls, cooldowns, timers, etc - many of these require no use of chat of voice comms)

- Picking on teammates by making consistently selfish plays to their detriment.(Courier stealing, going out of one's way to steal farm from a lower position teammate, unnecessary kill stealing)

- Improper role identification, your team strategically expects you to do X, you do Y. Y could even be better than X in terms of winning the game, it doesn't matter.

All of these above examples can either be common gameplay mistakes or intentionally malicious, but the point is that once your teammates do not trust one another, some will start verbally abusing, while others will begin to make similar mistakes as above (tilting) and lose the game for their own team.

Many players want to feel like they were the influencing factor that decided the game's outcome, and make choices that increase the influence they have on the game even if it might actually lower the chances that they win - and this is what many times leads to toxicity.

[ethbro]

Granted. And I'd be curious on the correlation between community toxcity and individual agency in competitive games. Possibly also game length.

It certainly feels like modern team games engender a different level of hate than, say, Quake III or Unreal Tournament.

Global matchmaking vs hosted servers probably haven't helped.

[shawn]

Sorry for calling you cocky.

[ethbro]

Np. I didn't take it as an insult. And I know what you meant -- I've seen the cool work that's come out of teams working at game shops.

And I hope I'm wrong, but I get the feeling they (and community managers) are working without the full support of their management to fix the problem. E.g. bandaid the broken bone

[shawn]

You're quite cocky. That's not an insult. I think if you cut your teeth on this problem for a couple months, you'd look back on this comment with some mixed feelings.

When a system is inherently toxic, there is no way to make it not-toxic short of coming up with a different substance. We can't make nuclear waste pure just because we wish it were. This is a very similar situation to this particular type of game. When it's a team game, and you rely on people who aren't doing their job, and you're separated by distance, it brings out the worst in you.

The problem isn't chat. If it were, it would be solved already, for the reasons you point out. The problem is the people. When someone doesn't like someone else, they will find a way to ruin their game while bypassing your censors. And yes, you can ban some of them, but not most of them, and not when most of the community acts like this. Which they do. Which you can't understand unless you go play the game for a bit.

[ethbro]

Toxicity is learned and reinforced (or not) behavior.

In the same way that HN comments are generally pretty constructive and charitable towards each other (and substantially more so than similar forums), so does negativity spread.

I'm not expecting any company to be able to turn their community into a paragon of empathy overnight. But I do believe that putting systems in place that punish poor behavior and reward good behavior has an impact in the long run.

From my first comment: > Therefore, companies simply aren't prioritizing [addressing the problem]. And won't, as long as the impact of toxic users isn't impacting the bottom line.

Because I honestly don't believe they care, in any non-monetary sense. They're essentially amoral. If they can make a billion dollars while people shout racist or sexist slurs at each other, they're fine with that.

And that's the primary annoyance I have with game companies. We hear "The player base is caustic, and we can't do anything about it, because it's too hard." When I believe reality is closer to "We ignored this problem as an industry (boy will be boys!), and now we don't want to be blamed for helping raise a generation of borderline sociopaths."

And yes, I spent many a college evening watching friends troll internet strangers in DotA, pre-LoL. So I'm familiar with the concept and execution.

[jack9]

> Toxicity is learned and reinforced (or not) behavior.

Not solely in a single game. That's some hubris.