You seriously can't think of a single idea for how to ban toxic players at scale?
Start with crowd-sourcing via reports, then pass it over to customer service once a critical mass has been achieved. Make sure one person can't easily create a new account after being banned. This really isn't a new problem, tons of companies of all sizes have tackled it with varying success.
Spoken like someone who’s never played the game. Get ready to be banned for no fault of your own, just because your three teammates decided to report you for the lulz. There are people who post on /r/dota2 complaining that they get banned from the game for nothing but picking techies each game. They don’t even say anything. People just report them immediately.
By the way, streamers are one of the most high impact users of the game. They are also one of the most unfairly targeted by report systems. This problem is extensively studied and very tricky.
I've been playing DOTA since the Warcraft 3 days, and a lot of HoN. I'm seriously grateful for the excellent Linux client.
Today I've moved over to Dota 2 and now I see that I no longer get banned and moved to unranked for picking techies. I play him at ancient/divine levels and getting reported is very common. Techies is my favorite hero as he is more about strategy than quick precise reactions. I think Valve has done some progress in this area, especially as you now can commend players.
Just because there are false positives or one company has a bad implementation doesn't make it an unsolveable problem.
As parent mentioned, ml sentiment analysis + logging + peer reporting + final human analysis = problem effectively combated
It's the final human analysis that companies are loathe to fund (e.g. Facebook), as costs scale with user count.
But it's not efficiently unsolveable with current tech. Therefore, companies simply aren't prioritizing it. And won't, as long as the impact of toxic users isn't impacting the bottom line.
Kay. What are the odds that some HN commenters are going to solve the same problem that has been an active priority for years of the people in the field?
I can’t speak for Valve or Riot, but at S2 it was a concerning issue. There’s just no good way to do it when the people involved are actively malicious. If you think there is, get ready to have your community collapse around you as everyone complains about unfair bans.
I don’t think you really appreciate the scale of the problem. Final human analysis is not possible when there are literally millions of games per week. It’s also not something that ML can identify cleanly — the moment it does, the culture will adapt to bypass the evaluator. It always does.
I absolutely appreciate the scale of the problem, and the adversarial nature of peer reporting. My day job has those same characteristics.
Millions of games per week * 30 minutes per game * avg lines of chat per minute = manageable w/ a proper streaming architecture
Especially when you have access to a massive, perfectly-scaled, distributed edge compute system. (i.e. running minimal, performance-optimized models on users' opponents' clients to do the initial detection / filter / compression pass)
But my point is this is fundamentally an economic problem, given current state of the art, not a technical one.
Companies are looking for pure-technical solutions because they're cheaper, and then complaining that it's a hard problem because they're unwilling to properly fund hybrid systems until state of the art can deliver.
ML is a first order approximation of human ability, not a magic unicorn that gives you exactly what you want. Thats the definitions of engineering: how do I build a system that fulfills my requirements from the pieces I have, not the pieces I wish I had?
So I don't feel much pity when companies allow toxic user bases to flourish because it's cheaper than funding solutions.
* Above intended in no way to belittle the awesome work folks are doing in the space with ML detection. But sometimes as engineers we need to admit when management is making unethical choices for financial gain
The chat is not the only way you can troll your team mates and cause them to rage quit (or scream at you and get banned themselves). I would actually say that the griefing created by people using slurs is actually pretty small.
You can be silent and ruin hundreds of games. Distinguishing that from genuinely bad players is not really easy.
How do YouTube, Facebook, and others ban illegal, unsavory content? Teams of moderators with good tools. Unfortunately, also a thankless job with the potential to give PTSD.
Youtube, Facebook and company have a ridiculous amount of false positives, for the money (and the people, they have quite a lot of human moderators) they throw at the problem. Also, they enjoy complete market dominance, so even if you want to move out of youtube... where are you going to go?
With the same rate of false positives there will be enough community rage (I guarantee someone is going to find a way to have your system ban the top 10 streamers of your game after 2 days you put it in place) your players (both good and bad) will go somewhere else. :)
The problem youtube, facebook and similar companies face is whole orders of magnitude more complicated than policing a game chat. It is, for example, highly unlikely that you will be censuring honest content when you encounter the words 'gas' and 'jews'.
You'd have to have the game invite only and do a interview with each applicant, checking their IDs and having them sign a contract after a psychological test. That's the only way, but of course, that's not possible; technically doable, but no, not possible.