[bbig3831]

There was a Washington Post article and arXiv paper on this topic several months ago. [1][2] The problem boils down to the definition of bias. Northpointe, the company that owns COMPAS, says the algorithm treats black and white defendants equally, which is true. Others claim that its predictions for blacks are wrong more often than for whites, which is also true. The main point of the paper is that you can't have your cake and eat it too. If you don't want to consider race, then the predictions will be wrong more often for blacks. If you don't want the predictions to be wrong more often for blacks, the algorithm has to consider race. (I'd recommend the articles for a better explanation.)

In my mind, the ownership of the software is what's troubling. First, it's not public, so nobody can look at the guts. And second, the Loomis decision constructs even more walls around the software. It's hard for people to critically examine and change something they can't see.

[1] https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1... [2] https://arxiv.org/pdf/1609.05807.pdf