[mikeash]

Punishment is always considered somewhat separately from the determination of guilt. The judge would already try to account for things like this when determining your sentence. They just do it in a deeply ad hoc and personal manner, where they just take a stab at it, try to account for things like how sorry you seem to be, apply guidelines, and come up with a number. This means that you might ultimately be punished for the judge not having a good breakfast:

http://www.scientificamerican.com/article/lunchtime-leniency...

And of course it goes without saying that judges will be affected by their biases, racial and otherwise.

I'm not sure what to do about it, though. Handing down the exact same punishment for every single person who commits a particular crime seems too blind. But any variation is going to be problematic.

[pessimizer]

All this does is systematize those biases so that they can't be challenged like a judge with a record of bias can. The statistics that they choose to record create bias in and of themselves - by using race in the algorithm, you are building in the possibility that race influences criminality. If you built in favorite foods, some foods would end up resulting in higher sentences than others, just as if you built in phases of the moon when the crime was committed or the astrological sign of the victim.

Where there was absolutely no effect, one out of every twenty combinations of all other variables would show significance in combination with the current value of that particular variable in the likelihood of future crime.

Furthermore, the algorithm would simply extend existing biases in arrest and sentencing, because it simply can't account for crimes that are uncaught and unpunished. Groups that are stopped, searched, arrested, and convicted at greater rates would without fail be sentenced to more time. Just another benefit of being white in America.

You end up using the fact that some groups are punished more often to justify punishing them more harshly.

Even worse, I bet that the fact that it thinks that women are at a higher risk for recidivism means that somewhere within the algorithm it's using the fact that women in general are less criminal than men to decide that women who do commit crime are more exceptional (within women), and therefore more deviant. It's disgusting. If you can't legally discriminate against a person on particular grounds, you certainly can't feed those grounds into an algorithm to let it discriminate for you while you shrug and feign innocence.

The algorithm is the innocent one - it's just attempting to reflect the system as it is. It's like an algorithm you would write to predict the winners of horse races, or the sports book. And just like one of those algorithms, if you stuff it with garbage (the kind of garbage that makes it wrong 77% of the time), it will result in garbage. If you use the results for something not external to the system, bad variables will feed back into themselves and make the results progressively worse - what's the effect of a longer sentence on recidivism? How does profitable is the arbitrage on your sports book algorithm if people use the results to bet, and the distribution of bets shift the odds?

[argonaut]

But they can be challenged. That's why you're reading an article about it. If you have a judge that is biased, it is probably harder to challenge his sentences than if you had an algorithm that you proved was biased.