[ZeroGravitas]

I wonder what the countermeasures to this are.

If you trained a model to predict the outcome purely from the protected class and it was successful (in terms of predictive power), does that mean fairness is efectively impossible?

e.g. if you trained an educational performance predictor on wealth of parents, then I'd guess it would do reasonably well. And there is the argument that your parents are rich because they're smart and you are genetically connected to them.

But there's obvious counterexamples, like children adopted by rich families or children of refugees (who may have been professors or surgeons in their home country).

So if we can't avoid the bias in that extreme example, then adding extra data is only going to bury that truth under confusion.

I'm not sure we're ready to admit that we disadvantage the children of the poor to ourselves, which will make this whole AI bias thing a tricky conversation to have.