[xyzzyz]

This approach doesn’t work. If sensitive features are actually predictive (and they very often are), the model will just learn to predict these features based on the non-sensitive ones. For example, if you know that someone is 30 years old, lives in Bay Area, and earns less then 30k/year, if you bet that this person is neither white nor Asian, you are more likely to win this bet.

[mathperson]

I don't like this example-they could be a graduate student

[xyzzyz]

Of course. They could be many other things as well. The question is, what would you bet on, if you didn't want to lose money?