[throwawaysocks]

> maximizing prediction accuracy is inherently unbiased

This assumes that you're actually maximizing prediction accuracy, rather than taking the easiest route toward sufficiently high predictive power.

Not-so-hypothetical: you can invest $N and create a profitable model that (unfairly and inaccurately) discriminates directly based upon race, or you can invest $N*M and create a profitable model that does not discriminate on race (regardless of whether its results are racially equitable). Given the choice, a lot of people will choose the $N approach.

[basseq]

> regardless of whether its results are racially equitable

Some would call this discrimination, if not outright racism.

As others on this thread have pointed out, at some point the algorithm follows the underlying data, which may also be discriminatory (zip code, etc.).