[gaius]

No-fly lists as well.

reinforcing trends we would rather diminish

Well, of course they are, at the end of the day all any predictive algo is doing is extrapolating a trend. It usually requires serious regulatory intervention - such as men being charged more for car insurance because "the computers said so" until legislation was passed barring sex as an input to the model.

[sdenton4]

The black box nature of the regression can make things difficult, though. If you have a couple other parameters which indicate 'black male' with high probability, you might find layer one reconstructing the notion of gender and layer two making decisions based on it.

[moultano]

Much better to include the feature in the training, and exclude it from the inference. That actually cancels out the effect, rather than just incentivizing the model to reconstruct it.