[codefreeordie]

At the risk of discussing sensitive topics on a platform ill-suited:

If, in your hypothetical recidivism case, an AI "accurately" determined that a pattern of higher recidivism-related features was correlated to race, and was able to determine "accurately" that the specific subset of recidivism-related features predicted race, why would it be wrong to make parole decisions using those recidivism-related features?

[pessimizer]

Because both the original conviction and any recidivism is determined through the decision-making of people who are aware of race and racial stereotypes. The AI would just be laundering the decisions you were already making, not improving them.

edit: imagine I was a teacher who systematically scored people with certain physical characteristics 10% lower than people who didn't have them. Let's say, for example, that I was a stand-up comedy teacher that wasn't amused by women.

If I used an AI trained on that data to choose future admissions (assuming plentiful applicants), I would end up with an all-male class. If this happened throughout the industry (especially noting that the all-male enrollment that I have would supply the teachers of the future), stand-up comedy would simply become a thing that women were seen as not having the aptitude to do, although nobody explicitly ever meant to sabotage women, just to direct them into something that they would have a better chance to succeed in.

[gadflyinyoureye]

If you decided on race, in this instance, you would be making people much more deterministic as a result of the power of race. Race is too broad a concept to reliably say that all white people are at X chance of recidivism. Instead we want to know if Marlowe is at risk of high recidivism based on her character.

[codefreeordie]

Both responses address the problem with a human making a biased decision based on race, which I think mostly we all agree would be bad.

The question I was posing is different, though, because this was discussing an AI system that looked at the underlying [in this case, recidivism] data which had race and race-adjacent information removed, and the AI has effectively rediscovered the concept of "race" by connecting it to some set of attributes of the actual [in this case, recidivism-predicting] features. If the AI were to determine such a link, that doesn't make its results biased, it just makes them uncomfortable. It's not clear to me that in such a case that would mean that we should remove those [recidivism-predicting] features from the dataset just because they ended up being correlated to race.