[xherberta]

As Cathy O'Neil points out, black-box algos are deciding things that have a huge impact on individuals' lives: which teachers get fired, which prisoners don't get parole, and which loan applicants get approved and denied. In the latter two cases, the algos are tougher on minorities.

It's troubling when we don't know why an algo is racist. We need ways of checking that these influential algos aren't reinforcing trends we would rather diminish.

[Houshalter]

No one cares about human judges that decide parole or to fire teachers. Humans are vastly worse. Attractive people are much more likely to get hired or get shorter prison sentences. Judges tend to give much harsher sentences just before lunch when they are hungry. Their predictions about what people are likely to reoffend are often worse than chance.

Additionally, the study about the "racist algorithms" was fraudulent. Their results were "almost statistically significant". I.e. not statistically significant. Compared to human judges which have well studied biases by race. There's nothing remotely fair about human judgement and you should always prefer an algorithm. In almost every domain they could find, researchers have found that even simple linear regression beats predictions of human experts.

And this has huge effects on our society. Humans making biased hiring decisions leads to mass discrimination against certain groups and very suboptimal employees. Having humans make loan decisions, means much higher interest rates, more people go bankrupt, and the economy grows much slower.

The EU banning the use of algorithms is just absurd.

[intended]

Yes, but Humans can go to jail.

You assume that your opponent is reality.

your opponent is other human beings. It takes precious little for a motivated person to learn how to hide malfeasance behind a black box.

Faith in un-corrupted black boxes should be considered the same way as faith in the un-coruptible internet

[Houshalter]

No one has any incentive to corrupt the black box. There are many, many cases where systems have terrible incentives that ruin everything. But I don't see how this is one of them. E.g. the bank has every incentive to make it's loan system as accurate as possible.

[xherberta]

This is a good point, and to her credit, Cathy O'Neil is careful to say that humans are worse. She's arguing for better algos, ones that are carefully thought out to avoid the worst of these pitfalls.

[p1esk]

I think he was complaining about the algos' decisions being hard to understand, not them being unfair. From your examples, it seems like we more or less can understand human decisions.

[gaius]

No-fly lists as well.

reinforcing trends we would rather diminish

Well, of course they are, at the end of the day all any predictive algo is doing is extrapolating a trend. It usually requires serious regulatory intervention - such as men being charged more for car insurance because "the computers said so" until legislation was passed barring sex as an input to the model.

[sdenton4]

The black box nature of the regression can make things difficult, though. If you have a couple other parameters which indicate 'black male' with high probability, you might find layer one reconstructing the notion of gender and layer two making decisions based on it.

[moultano]

Much better to include the feature in the training, and exclude it from the inference. That actually cancels out the effect, rather than just incentivizing the model to reconstruct it.