[ohthehugemanate]

Bias is a pattern-generation process. Machine learning is a pattern-recognition process. Any bias on the part of the (human) data collection, or the (human) training program author, gets spit out as a "pattern", because it is one. The problem is that it gives the illusion of a bias in reality.

My go-to example is machine learning police enforcement direction, often used as a counter to racially biased policing. This works in any city with a historical problem of racial bias in policework. We give the algorithm all the data we have from the last 60 years of policing this city. Patrol schedules, incident records, arrest records... everything. The computer magically tells us where we should focus our efforts. To the police chief who paid for the system, and especially to the media reporting on it, it looks like a computer is making the decisions without bias. Hooray!

Of course, anyone who's ever worked with machine learning can spot the problem. The data set was generated by racially biased policing. That bias will be reflected in all the records: more arrests for race X, more patrols scheduled through their neighborhoods, more incident reports from those areas. So when the algorithm says "increase patrols in this neighborhood," or "look for people who fit this profile," it is simply synthesizing the patterns from 60 years of racial bias. So the police in LA have a real problem: their "unbiased" computer program is telling them that their criminals look like black people, and they should increase patrols in Compton. So they do, and that data only takes the data further from "un-biased" reality. In fact, the police "black box" is only pointing out a history of racially biased policing. We're relabeling it as recommendations for future behavior.

[yummyfajitas]

Of course, in reality this bias can be corrected for. I don't know if specific crime-stat software does it, but it's certainly doable. Here's an example where I solve literally the same problem (better measurement => more events detected) in a different scenario. Is it remotely controversial that I can do this for a sensor network?

https://www.chrisstucchio.com/blog/2016/delayed_reactions.ht...

You might also be interested to know that a variety of studies have shown that policing is not particularly biased. Arrest statistics and the like correspond pretty well with NCVS and similar crime victim surveys.

http://slatestarcodex.com/2014/11/25/race-and-justice-much-m...