[kenjackson]

(From my above reply too, as it applies here also):

Lets be clear -- if the null hypothesis in this case is true (that there is no bias), and all other assumptions made are true, there is a slightly greater than 5.7% chance of obtaining this result (or something even more skewed). That's a great bar for publication of SCIENCE. It's not a great bar for hiding behind a proprietary algorithm used in sentencing. People talk about misuse of p-values, but this takes the cake.

[yummyfajitas]

If you want to criticize the details of her analysis, go ahead. I'm solidly in the Bayesian camp and I agree with you 100%. What I'd have done is computed posteriors on all these coefficients and then computed bayes factors/probability of bias.

I'm confused though; the mood affiliation of your post somehow suggests that her less than perfect choice of a statistical methodology somehow supports her claims. Could you explain that? Or am I simply misunderstanding what you are trying to say?

Also, lets suppose we just take her own analysis at face value, and don't view it through the p-value lens. The maximum likelihood estimate suggests that even if this effect is not random chance, it's not very big. I.e., the "score factor high" estimate is >8x larger than the "score factor high, race = black" estimate. Isn't this really good? Do you really think the human biases that this algorithm mitigates are lower than this?

Lastly, what specific analysis would convince you that this algorithm is predictive and non-biased (or more realistically, not very biased)?

[pdkl95]

> maximum likelihood

That may be grounds for a mistrial. Decisions about crimes are not judged by the "maximum likelihood".

> what specific analysis would convince you that this algorithm is predictive and non-biased

What is it going to take to convince you that the choice of model and which data to use as input is just as important as the analysis itself?

> race_factor

Depending on the situation, using race or other protected classes is illegal. One of the reasons we have a right to face our accusers is to provide an opportunity to challenge those accusations. Racial (or any other protected class) discrimination doesn't become legal when it is hidden behind an equation or algorithm. If the government wants to keep the method secret, then anything derived from those methods should be excluded.

> human biases

...are off topic. An algorithm needs to justify it's own existence.

> it's not very big

So you're fine with racial bias, as long as it only affects what you consider a "small" number of people.

> or perhaps black defends actually are more likely to commit crimes

/sigh/

[yummyfajitas]

What is it going to take to convince you that the choice of model and which data to use as input is just as important as the analysis itself?

I'm already convinced of this. Are you trying to imply that the cox model is wrong or something? If so, why not just make that argument explicitly?

Of course, if the Cox model is wrong, why do you believe the algorithm is biased? Isn't that reason to disregard the entire ProPublica article (which is all based on the Cox model)?

Depending on the situation, using race or other protected classes is illegal.

Did you even read the article? "Northpointe’s core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions. "

/sigh/

You can emote all you like. Reality does not change.

I must admit, the emotion on display here confuses me. Much like you I oppose racial bias. The R script provides evidence that very little racial bias is present in this system. Why does this inspire such negative emotion? It's almost as if you care more about looking anti-racist than you care about having racism's effects be reduced.