[dohnuts1919]

Agreed. I can't remember the details but I was listening to a podcast lately (Sam Harris maybe?) where the guest was talking about an example of an ML-driven system that was being used to determine whether or not to approve mortgage applications, and upon closer inspection it turned out to be latching onto some data that gave clues as to the applicant's race and using that in its decision.

You can guess the outcome: the software re-invented redlining.