[Fomite]

What's infuriatingly ignored is that in that very same PLoS Medicine issue is a response to Ioannidis' work by Greenland, IIRC, that notes that by "False" he means the significance is wrong, but what's really of interest is the effect measure.

On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

[capnrefsmmat]

It's true that effect sizes are often more important, but it's also true that they're also often incorrect. See e.g.

Ioannidis, J. P. A. (2008). Why Most Discovered True Associations Are Inflated. Epidemiology, 19(5), 640–648. doi:10.1097/EDE.0b013e31818131e7

Most studies are underpowered and are incapable of detecting the true effect. Only if they get lucky and observe an abnormally large effect will they obtain a statistically significant result, so the published results tend to be significant overestiates.

For another good example, see

Gelman, A., & Weakliem, D. (2009). Of beauty, sex, and power: statistical challenges in estimating small effects. American Scientist, 97, 310–316.

http://www.stat.columbia.edu/~gelman/research/unpublished/po...

[Fomite]

I think part of the point there is not to pass effect estimates through a significance test filter first. Most studies are underpowered to detect a true effect at alpha = 0.05. That doesn't actually suggest that most studies are wrong as much as if a study is underpowered and doesn't find a significant finding, we assert its dull and uninteresting.

Ironically, the Ioannidis paper is in Epidemiology, which is a journal that is fairly anti-significance testing, but where I still get reviewers suggesting that an effect measure with a confidence interval that brushes against the null must mean nothing at all.

[tokenadult]

On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

This is a fair question. I think the reasons the Ioannidis paper was persuasive are that

1) Ioannidis replicated earlier results about the lack of replication of most research reports,

and

2) Ioannidis "showed the work" for how possible, and indeed likely, it is for an effect size that permits a false-positive finding to be published, under reasonable assumptions about the prevalence of false-positive findings and publishing practices. Most scientists were vaguely aware of lack of replication years before anyone heard of Ioannidis, but not many scientists were fully aware of how readily a false-positive finding can be published.

[eli_gottlieb]

>On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

Because in science, not believing things is the default state. If you say, "most published findings are false", you're really saying, "most of the time we have to accept the null hypothesis", which is what we all not-so-secretly believe regarding everything, all the time, in any case.