Hacker News new | ask | show | jobs
by nshepperd 3994 days ago
A significance level says very little, if anything, about the probability of getting an "insignificant" result given that the null hypothesis is false. That's all about statistical power. It's entirely possible that those "5 against" were just underpowered (sample size too small). Or that the sample sizes were systematically too small, and those 5 just happened to be the ones that failed to reach statistical significance. Or that the study was actually well-powered and the difference really is localized geographically.

Either way, a 95% significance level by itself tells us little about how we should interpret those 5 against.