Hacker News new | ask | show | jobs
by harterrt 2107 days ago
Unless I'm reading your comment wrong, this p-value (0.03) is actually significant for a 95% confidence test.
2 comments

0.03 is the odds ratio (a measure of effect size, an odds ratio of 1 would mean the treatment and control arms had the same rate of ICU admittance). The p value is 0.08 from here "CI:-0.30 - 0.03 p:0.08." which captures the likelihood the treatment had an effect.

That said, just looking at p values and applying a cutoff at 0.05 is pretty bad practice that is getting a lot of heat thanks to the replication crisis (does it make sense to behave as though p=0.08 is not true and something at p=0.049 is true? almost certainly not). If you get a value in this range and a huge effect size then it's a really good idea to repeat the experiment with way more data. It's also a common stats error to act as though p>0.05 is the same as knowing something DOES NOT work, all you can say is this specific study wasn't able to show that it does work with 95% confidence.

That's part of the confidence interval. The adjusted p-value is 0.08. However, the cut-off of 0.05 is just a convention. https://en.wikipedia.org/wiki/Misuse_of_p-values. I'd think of this as a grey scale, where 0.08 is roughly in the significance range.

The null hypothesis is that it's highly unlikely VitD has an effect, and we should expect to see that substantiated often in tests. How often? 95% of the time. 5% of the time we can expect to see spurious results from our simplistic model of random processes. Upshot, it's a small change to move those numbers to 92% vs. 8%. In this context, it's fine to say "this was a small pilot that directionally shows we should do a much bigger test", which is what they're now doing.