[geofft]

This seems essentially like Searle's Chinese room. The program is not biased, in the sense that there is no line of code that adds bias, but the application of the program to the available data clearly produces a biased process.

We've seen this exact thing with other machine learning algorithms - there was the one in the news recently that insisted on classifying a photo of a man in the kitchen as a woman, because all its training data firmly convinced it that women are the only people in the kitchen.

I guess the thing worth explicitly asking is whether biases for entirely logical reasons are defensible - for instance, if you start with an industry where (for whatever reason) men are paid much more highly than women, it's okay to offer people their current salary + fixed delta to poach them from their jobs. I would say no, because the goal of legal policies like this is to achieve a specific result in society, and specifically not to punish bad people in charge of companies, so the question is not whether people had a bad motive, but whether the result is being achieved. If you're allowed to apply a non-biased algorithm to biased starting data and have it yield an equally-biased output, you're not actually solving the bias.

[dogruck]

What if we observe differences in said algo(s), when applied to small companies and the Googles?

Said another way, if Google's employee stats are biased, is there any feasible defense?