Hacker News new | ask | show | jobs
by jamesaguilar 4607 days ago
Probably false? As in you have a better chance claiming the negation of a scientific paper's conclusion than the actual conclusion? I doubt it.
3 comments

In certain fields, that's not implausible. Many results are along the lines of "compound X is effective against disease Y", where the negation, "compound X is not effective against disease Y", is a reasonable baseline assumption, because most compounds are not effective against most diseases.

Results where the prior odds could plausibly be considered 50/50 are another story. Scientific research that doesn't take the form "rule out the null hypothesis with p values" is also a different matter (lots of CS and physics, among other fields, has a more complex mingling of theory and epistemology than experiment/nullhypothesis/pvalue/repeat).

That's a good point. I didn't consider the fact that possibly many discoveries have a very low prior expectation of truthood.
I wouldn't be so sure. Not talking about mathematics or CS, but in many social sciences (and also medicine and so on) the paper is just stating that A->B but i) there can be a lot more things going on that explain whatever correlation you are finding (from reverse causality to bad experimental setup), and *more importantly ii) in order to get published, you need to present a somewhat interesting or controversial statement.

If you state something obvious and most likely true, good luck getting to Nature/Science. If you state something unusual, then that will sell (where selling is getting citations) so you will see it published

Asimov wrote an excellent piece on this - The Relativity of Wrong: http://chem.tufts.edu/answersinscience/relativityofwrong.htm

The Earth isn't flat, but it certainly approximates being flat for small measurements of its surface (such as those early civilizations would've been able to make).

yes, probably false. This is not news, e.g. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ came out in 2005.

of course, research "proving" that research tends to be false can get us into a little circular game...

There's a follow-up paper to the one you linked that claims that much less published research findings are false:

>We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, P=0.18) or with respect to journal submissions (0.5% more FP per 100 submissions, P=0.12).

[1] http://biostatistics.oxfordjournals.org/content/early/2013/0...

The circle has begun :)

You should also read the responses to that article, such as Ioannidis's, which is scathing. They're all open-access, thankfully, and you can find them here:

http://simplystatistics.org/2013/09/25/is-most-science-false...

Jager and Leek put their code on GitHub, so the commenters were able to review their code and tinker with it to see what would happen.

Thanks for all these links! I skimmed them, and it's fun to see that two of them rip into Ionnadis' work, while Ionnadis rips into the Jager/Leek work... It's also a cool exercise in reproducible research and open peer review, both of which are far from common.

In my opinion, the entire exercise of data-mining the published literature is pretty much futile. We already know there are problems in the published literature and that scientists are pretty mediocre at statistics. Pin-pointing the exact value of how mediocre only leads to, as your link shows, a mountain of published works, hurt feelings (on Ionnadis side I guess) and doesn't solve anything.