| I remember ten years ago at least some AI systems were done on loan approvals, and bam, and out came the same structural racism in financing and loans. If the number of variables is low, then the extant bias may be cooked into the inputs and the AI result is then inevitable. For example, income will be suppressed all other variables being the same, that's an immediate loss in approval chances. If crime maps and the like go into some risk profile on the safety of the property, well guess what, that'll be effectively racist and/or lower loan preapproval. It doesn't have to just be that the race checkmark/dropdown becomes an input. But you're right, the AI is a black box. Rules engines or other calculations at least can be back-traced, and maybe counter-weighted. A great deal of structural and persistent racism comes down to housing: where you live. It doesn't take a die-hard Zillow user to know that where you live will signal things to the loan application algorithms, regardless of "race blind" applications if that is even a thing. Thus, higher loan rate, lower maximum value, less access to various stratifications of neighborhood "eliteness". And the transaction rate/speed on housing is really low (annoyance to move, realtor fees, closing costs, PITA to shop for new/sell your old), so policy to address the bias would take a longer amount of time to actually show up in the stats than its political survival time (because so many well monied interests will lobby for its removal). And thus, the more things change, the more it just stays the same. |