[grinnbearit]

That doesn't makes sense, the only reason race would matter is if we're not using all other available information

For example, imagine if income perfectly explained default rates. Then in that case, the race of the person wouldn't matter at all given income.

P(default | race, income) = P(default | income)

The only time this equality would be false is if race was being used as a proxy for another variable that isn't being collected.

Imagine If incomes were lower for certain races, in that case the algorithm would be biased in favour of those discriminated people, not against them.

P(default | discriminated race, income = X) < P(default | ~discriminated race, income = X)

this article goes into more detail https://www.chrisstucchio.com/blog/2016/alien_intelligences_...