Hacker News new | ask | show | jobs
by i_phish_cats 2643 days ago
I predict nothing will change. Flaws in p-values and confidence intervals have been apparent since almost their inception. Jaynes spoke out against it strongly from the 60's on (see, for example, his 1976 paper "Confidence Intervals vs Bayesian Intervals"). Although I can't find it right now, there was a similar statement about p-values from a medical research association in the late 90's. It's not just a problem of misunderstanding the exact meaning of p-values either. There are deep rooted problems like optional stopping which render it further useless.

The problem is that with all its problems, statistical significance provides one major advantage over more meaningful methods: it provides pre-canned tests and a number (.05, .01, etc) that you need to 'beat'. The pre-canned-ness/standardization provides benchmarks for publication.

I once worked in a computational genomics lab. We got a paper into PNAS by running fisher-exact test on huge (N=100000+) dataset, ranked the p-values, got the lowest p-values, and reported those as findings. There's so much wrong with that procedure its not even funny.

2 comments

Hippocratic medicine lasted well into the 19th century, centuries after the scientific revolution. There'd been critics correctly calling it an intellectual fraud before then. You could've taken this as proof that no force on Earth could drag medicine into modernity, but it did sort of happen, as it became public, common knowledge that doctors were harming more people than they helped. They did start cleaning up their act (literally) though it took a long time and I think they're still collectively irrational about chronic conditions.

I hope we aren't worse at reform than they were in the 1800s.

Working in the field, it is getting better. It's slow, but getting better.