[backpropaganda]

Can you give an example of a situation where an ML application would be reinforcing a problematic bias but still have good performance metrics? My point is that a wrongly-applied ML application would suffer in just plain accuracy. For instance, a Automatic Carrier Counsellor might give "homemaker" as a suggested career choice to women, but then before we start calling it biased, it would already be wrong. If the same algorithm had dug deeper, it would have learn that the said woman would be a great programmer.

[praxulus]

Recidivism prediction systems will usually tell you that black people are more likely to get arrested/convicted again. They do so accurately, but also result in longer sentences for black people.

https://arxiv.org/abs/1610.07524

[xupybd]

Yeah but doesn't that have more to do with the way the predictions are used?

It seems to me to be a stupid thing to do. This person seems more likely to get convicted again, lock 'em up longer. Instead of asking why is this person more likely to get convicted again? Can we prevent this in a redemptive non punitive way?

It's really useful to have that prediction/data but how you use it is more important

[ZenPsycho]

the problem is a layperson doesn't necessarily know what a prediction necessarily means without a deep understanding of how the system is making its predictions, let alone how to apply it.

worse is that since the prediction is coming from computer that lends the prediction an air of authority another article called "bias laundering". the general belief is that computers are objective and cannot have bias, which in a sense is true, but people don't tend to think a step further about the problems and biases in the people who programmed the computer.

so that is definitely a thing usually missing from these discussions is that the people using these systems generally don't know how they work, and believe they predict or imply things that they don't

[industriousthou]

I mean, that same algorithm could be used to determine that blacks or other at-risk groups should receive extra attention or support. An accurate picture of reality can be used poorly or well.

[benchaney]

There is a much more fundamental problem, which is that people are bad at understanding the difference between "is" and "should". No amount of information about what the world looks like tells you anything about what course of action is the most moral (and vice versa). If you are building a system that predicts recidivism rates (figuring out what "is"), then any piece of information that improves your accuracy is good. If you use that system to suggest sentences (making decisions about "should), then you are going to run in to a lot of problems.

[industriousthou]

I still don't understand why information should be elided from judges during sentencing. If public officials use data to worsen issues like recidivism rather than improve them, then those officials should be removed. If a judge can't be trusted to act responsibly and morally with accurate information about a defendant then why would we even begin to think that they're competent?

It's just the reality of how the justice system works. We have trust in the approximation of justice that the judiciary provides and constantly struggle to improve that judiciary.

[throwanem]

That sounds like exactly the kind of thing you'd expect to happen when you treat people as feature clusters instead of, you know, people.