[raceyT]

Are you saying that if one uses a criterion that incidentally correlates with race that is racism? And wouldn't a correction for this itself cause a much more disturbing racist exercise ie; profiling every possible data point against race and forever maintaining such profiles?

[CPLX]

If a bank made a decision not to lend to people that subscribed to Spanish language HBO, would you consider that to be legal? Ethical?

How about if they wrote software with a single criteria for loans, and that criteria was "don't lend to black people"?

Right, we know that's not OK intuitively. What about an algorithm that automatically declines all applications from people who went to historically black colleges?

So then, where exactly do you draw the line? If said algorithm is opaque does that make it OK? What if it wasn't intentional, but the algorithm just kind of turned out that way by machine learning? Is that better somehow? And how can we tell, and how can we hold the bank accountable for those decisions?

One answer to "where do we draw the line" is to say that we only make credit decisions based on history and experience with actual credit. Things like payment histories and so on, and that the both the criteria and the data used to make those decisions must be transparent, and consumers have the right to challenge it.

This, in fact, is the system we currently have. There are reasons for that.

[raceyT]

I would consider it not only reprehensible but absurd, as these examples are.

I can't think of any advantages a racist bank would have, can you? The approach by this lender is rather to dig deeper into individuals and give them an opportunity based on personal data. It seems just as likely that resulting correlations could help any given race.

I believe algorithmic attempts to understand people are a form of intelligence to support, even if they discover things we might not like or make mistakes along the way. Should they find uncomfortable truths we can own up, have a laugh and grow; it will be a lot easier than fighting facts.

Prejudice is literally pre-judging, making assertions based on fear without knowing the facts. Perhaps it's the algorithms themselves that are in need of protection here.

[flashman]

> I can't think of any advantages a racist bank would have, can you?

Banks can, which is why redlining was a thing: https://en.wikipedia.org/wiki/Redlining