[gadflyinyoureye]

If you decided on race, in this instance, you would be making people much more deterministic as a result of the power of race. Race is too broad a concept to reliably say that all white people are at X chance of recidivism. Instead we want to know if Marlowe is at risk of high recidivism based on her character.

[codefreeordie]

Both responses address the problem with a human making a biased decision based on race, which I think mostly we all agree would be bad.

The question I was posing is different, though, because this was discussing an AI system that looked at the underlying [in this case, recidivism] data which had race and race-adjacent information removed, and the AI has effectively rediscovered the concept of "race" by connecting it to some set of attributes of the actual [in this case, recidivism-predicting] features. If the AI were to determine such a link, that doesn't make its results biased, it just makes them uncomfortable. It's not clear to me that in such a case that would mean that we should remove those [recidivism-predicting] features from the dataset just because they ended up being correlated to race.