[AnimalMuppet]

But you can have distortions in the data, even if the algorithm is neutral, can't you? The data says "men are more likely to commit rape than women"; OK, that's probably not just that the data encodes a bias. But if your program says "blacks are more likely to be charged with violent crimes", say, is that because blacks are more likely to commit violent crimes, or because blacks are more likely to be charged with violent crimes because the justice system is (or historically has been) skewed?

Even an unbiased analysis system can reach bad conclusions from bad data, and a biased justice system can produce bad data. So the conclusions can be biased even if the program is unbiased.

[yummyfajitas]

Yes, I agree that not all learning algorithms are perfect.

Is it your belief that we can cook up better algos/data collection methods/etc and all the people complaining about "bias" in algorithms will be satisfied? I don't believe that is the case, given that no one is actually complaining that the algorithms are getting the (factually) wrong answer.

[AnimalMuppet]

My assertion was that a perfect algorithm, fed biased data, produces biased results. Given Ferguson, etc., data from the justice system may be biased, at least for some locations.

If the arresting officers are biased, the number of arrests are probably biased. I'm not sure that you can fix that with better data collection methods; the problem isn't with the data collection. The problem also isn't with the analysis.

It may be possible, at least in theory, to create an algorithm that will determine whether the officers are biased, or whether the race in question actually commits more crimes. I'm not optimistic about that working in reality, though.