[ramanujan]

This paper has gotten way too much press for an oversimplified model of science. Here's the thing: if results hold up to scrutiny, the authors are eager to share code and plasmids/samples. If not, they are a lot more squirrelly. Outside replication is what keeps the machine moving forward, is fairly readily proxied by citation rates, and yet is not captured by Ioannidis' simple model.

[samd]

That's not the case, even the most widely cited research is dubious. From the Atlantic article:

"He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association."

[izendejas]

Were some of those citations made to prove those findings wrong? Dumb question, but would be a shame to not be thourough when attacking bad literature.

[_delirium]

Yeah, that's a commonly mentioned problem with citation-counting. Much like linkbaiting on the internet, a poor-quality paper taking an inflammatory position can get a lot of citations from people debunking it. Another problem is throwaway citations: some paper gets cited as a generic example, rather than because it provides anything valuable that the paper citing it actually draws on.

Unfortunately, it's much harder to come up with better measures. Given a smallish corpus of a few hundred papers, humans could read through them and annotate each citation with things like, "cited to debunk", "cited to distinguish related work", "cited for general background", "cited in passing", "cited for result", etc. But computers are not yet very good at doing that automatically, so the large-scale citation analysis just does dumb citation-counting.