| I think they mean algorithmic as in, simpler and deterministic and auditable plain if/then/math algorithm, vs black box magic AI. Even if an AI can produce seemingly the same results as a human, it should be out of the question anyway to let an inscrutable black box make decisions over people's lives. Because at least with a human you (their boss, or a judge, etc) can ask them "Why did you decide that?" and they can tell you. A racist or mysoginist or religious human can be identified and fired or corrected etc. How do you judge if an AI is giving inhumane decisions? The decisions themselves can't really be judged, only the process that generated them, and you can't see that process in an AI. If someone doesn't get a loan, and someone else does, you can't tell that wasn't right just from the final result. Even if the results "look" wrong, like only 30% of black people get the loan while they made up 40% of applicants, even that could possibly be exactly correct, but you can't know if you can't see the process. But a human can be asked, and simple algorithm code can be read. Probably these days the human has so little discretion anyway that the corporate policy is the algorithm and the human is pointless anyway except as a sham human-looking interface to appease customers. It helps sales to have a human, but the human in fact wields none of the human power that the customer wants a human for. |
If the number of variables is low, then the extant bias may be cooked into the inputs and the AI result is then inevitable. For example, income will be suppressed all other variables being the same, that's an immediate loss in approval chances.
If crime maps and the like go into some risk profile on the safety of the property, well guess what, that'll be effectively racist and/or lower loan preapproval.
It doesn't have to just be that the race checkmark/dropdown becomes an input.
But you're right, the AI is a black box. Rules engines or other calculations at least can be back-traced, and maybe counter-weighted.
A great deal of structural and persistent racism comes down to housing: where you live. It doesn't take a die-hard Zillow user to know that where you live will signal things to the loan application algorithms, regardless of "race blind" applications if that is even a thing.
Thus, higher loan rate, lower maximum value, less access to various stratifications of neighborhood "eliteness". And the transaction rate/speed on housing is really low (annoyance to move, realtor fees, closing costs, PITA to shop for new/sell your old), so policy to address the bias would take a longer amount of time to actually show up in the stats than its political survival time (because so many well monied interests will lobby for its removal).
And thus, the more things change, the more it just stays the same.