[kenjackson]

Lets be clear -- if the null hypothesis in this case is true (that there is no bias), and all other assumptions made are true, there is a slightly greater than 5.7% chance of obtaining this result (or something even more skewed). That's a great bar for publication of SCIENCE. It's not a great bar for hiding behind a proprietary algorithm used in sentencing.

People talk about misuse of p-values, but this takes the cake.

[mdjeu]

This is in my professional area, and yummyfajitas is right on certain points. The reason these approaches started taking off at all is because the alternative, subjective decisions, don't generally work as well. There's plenty of meta-analyses to show this; that's why these risk systems get used.

Also, this analysis is certainly a useful addition to the literature on this system, but it's one analysis, and regardless of your philosophical stance on p-values, a p-value of .057 in the presence of multiple testing isn't the most convincing thing.

Having said that, the use of non-open predictive systems is a problem for criminal settings. Maybe this thing is biased, but the only way to find out and fix it is to do these sorts of analyses and have this sort of discussion.

The problem isn't the use of prediction systems, it's the use of them without open academic scrutiny, without correcting any biases that emerge.

[kenjackson]

but it's one analysis, and regardless of your philosophical stance on p-values, a p-value of .057 in the presence of multiple testing isn't the most convincing thing.

I agree in general. But when you have one data point and it relates to bias in a system a p-value of .057, suggesting there is bias is more compelling than the null hypothesis. Especially when other independent a-priori evidence seems to also point against the null hypothesis.

[yummyfajitas]

I replied over there: https://news.ycombinator.com/item?id=11756449