[Eridrus]

The biggest issues of bias/fairness in ML are not to do with the algorithms or results, but the underlying data.

A trivial example would be: what if you trained a classifier to predict whether a person would be re-arrested before they went to trial? Some communities are policed more heavily so you would tend towards reinforcing the bias that exists and provide more ammunition to those arguing for further bias in the system, a feedback loop if you would.

Or what if some protected group needs a higher down payment because the group is not well understood enough so that you can't distinguish between those who will repay your loans and who won't? Maybe educational achievement is a really good predictor on one group, but less effective on another. Is it fair to use the protected class (or any information correlated with it) when it is essentially machine-enabled stereotyping?

Recently it has been noted that NLP systems trained on large corpuses of text tend to exhibit society's biases where they assume that nurses are women and programmers are men. From a statistical perspective this correlation is there, but we tend to be more careful about how we use this information than a machine. We wouldn't want to use this information to constrain our search for people to hire to just those that fulfil our stereotypes, but a machine would. This paper has some details on such issues: http://arxiv.org/abs/1606.06121

I don't think there are any easy solutions here, but I think it's important to be aware that data is only a proxy for reality and fitting the data perfectly doesn't mean you have achieved fair outcomes.

[h4nkoslo]

These are not "fairness" issues; these are process feedback issues. The same problem pops up if you're using algorithmic selection of machine parts to test for failure, attempting to programmatically evaluate patches for code quality, writing fraud detection algorithms, etc.

[bo1024]

Great points. These are exactly the kinds of issues researchers are grappling with in trying to nicely define, and attempt to achieve, fairness in ML.

[sevenless]

One of the great things about programming a computer to do something is that it forces you to expose and make explicit every hidden assumption. "Computers are very stupid: they only do exactly what you tell them to do."

With biases about people based on immutable characteristics (sex and race), we need to be clear about why stereotypes are bad and what we hope to achieve by eliminating stereotype-based reasoning. There is a great deal of hypocrisy and pretense around this subject, but only by being explicit and unapologetic can we explain to a computer what it is we want to achieve.

Stereotypes are not bad because they are false. Many stereotypes, even negative and unpleasant ones about vulnerable minorities, are statistically true at this time. A stereotype is nothing but a certain kind of model, and indeed, models built on sex and race stereotypes may perform better than those that aren't.

Nonetheless, we have strong norms against using stereotypes in law, public life and employment, because the outcome of such reasoning would be intrinsically unjust, and because of a long history of political struggle against a society, that explicitly discriminated on the basis of race, sex, homosexuality and so on. Conservatives will disagree with these premises, but we implicitly reject a conservative, discriminatory vision of society. Rejecting oppressive and unjust stereotypes is an unavoidably political act.

We recognize that

[0] It is a category error to treat humans like other kinds of objects which can be measured, because human beings are intelligent and can alter their behavior. Telling humans that science has discovered certain facts about their behavior may well change their behavior, or even re-order society around these new 'facts'. There is no neutral ground; doing statistics on humans has ethical implications. An awful example of this was the eugenic movement that inspired the Nazis.

[1] Stereotypes can reify themselves. A society which treats women as less than men will end up as a society where women are less than men, and are systematically harmed - they will be less educated, and will get treated as less intelligent. This is a kind of positive feedback loop between the widespread endorsement of a stereotype and its being 'confirmed as true'.

[2] It's intrinsically unjust to judge individuals, especially in a negative way, based on the behavior of others. This is a matter of justice, and it overrides considerations of predictive accuracy.

[3] We live in a society that is still unjust, racist, sexist and so on. We want a society where fair and equal opportunities are given to everyone, and where everyone has a chance to escape from negative stereotypes. This overrides efficiency. In that sense, we can hope to change the nature of society by changing the nature of claims that are widely held as 'truth'.

These are sophisticated premises for rejecting stereotype based reasoning, and they come from rejecting the society-wide outcomes of treating stereotypes as truth. We know what these effects are, because we know what a discriminatory society looks like. But this is the kind of reasoning we can use to build non-discriminatory, socially just models that do not harm people simply for being who they are.