[dogruck]

What about a more socially sensitive domain, such as college admissions, hiring, or setting pay? What if you put such an algorithm on said task -- to avoid human bias -- and later observe that the algo, say, does not hire <pick your group>?

[maksimum]

Instead of Institutional Review Boards approving experiments we could have Algorithm Validation Boards. Hold out a portion of your dataset for validation, and have the algorithm designers examine the outcomes in communication with ethicists, lawyers, and business strategists.

I don't think the two hands can properly "validate" algorithms without communicating. The algorithm designer can maximize AUC, but what if one <group>'s class is 95% label A; the designer always predicts A for <group>. How bad is ALWAYS missing 5% for label B? If you can put a price on it, then the developer can build it into the algorithm. But if the price is difficult to accurately estimate, or non-monetary qualities are desirable, it may be hard to build them into the classifier ahead of time. On the other hand if the cost of perfect <hard to quantify criterion> reduces AUC significantly, algorithm designers need to communicate that...

[andreareina]

That's already happening. Can't remember which article to link to, but machine learning algorithms make decisions that reflect pre-existing biases (e.g. harsher sentences for black people).

Humans produce the data that the ML algorithms train on, so without whitening the data somehow (and there's another contentious issue) we should expect the result to be biased as well.