[timr]

"Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal."

That's just incredibly unfair. There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the primary reason that papers aren't reproducible is just noise and basic statistics.

As a scientist, you control for what you can think of, but there are often way too many variables to control completely, and it's probable that you miss some. Those variables come to light when someone else tries to work with your method and can't reproduce it locally. However, real scientists don't stop and accuse the original authors of being "unprincipled" -- nine-point-nine times out of ten, they work with the original authors to discover the discrepancies.

It isn't surprising at all to actual, working scientists that most papers are impossible to reproduce from a clean room, using only the original paper. It's the expected state of affairs when you're working with imperfect, noisy techniques, and trying tease out subtle phenomena.

[TeMPOraL]

> It's the expected state of affairs when you're working with imperfect, noisy techniques, and trying tease out subtle phenomena.

Sounds like a ridiculously low standard. If your paper is in principle unreplicable, then I only have your word for evidence of what you're claiming. This is not science. Even journalists are held to a higher standard.

[michaelt]

  There are some fields and methodologies where p-hacking
  and cherry-picking have been a problem, but the primary
  reason that papers aren't reproducible is just noise and
  basic statistics.

It's possible to imagine a version of academia where results that can be attributed to noise don't get published.

[wyattpeak]

Is it? That doesn't seem at all obvious to me. In fact it seems decidedly impossible.

Almost any result could in principle be attributable to noise; where are you planning to source all of the funding to run large enough studies to minimise that? And no matter how large your experiments or how many you run, you're still going to end up with some published results attributable to noise since, as GP says, that's the nature of statistics. By its nature, you cannot tell whether a result is noise. You only have odds.

I'm not saying there aren't problems with reproducability in many fields, but to suggest that you can eliminate it entirely is naive.

No, not naive - wrong.

[TeMPOraL]

> Almost any result could in principle be attributable to noise; where are you planning to source all of the funding to run large enough studies to minimise that? By its nature, you cannot tell whether a result is noise. You only have odds.

Well, with a single paper the odds indeed are that it's noise. That's why we need reproduction. Now of course a paper needs to be published for it to be replicated later. But the paper (and/or supplemental material) should contain all possible things the research team can think of that are relevant to reproducing it - otherwise it's setting itself up to be unverifiable in practice. Papers that are unverifiable in practice should not be publishable at all, because a) they won't be reproduced and thus it'll be forever indistinguishable from noise, and b) there's no way to determine whether it's real research, or a cleverly crafted bullshit.

[wyattpeak]

I don't disagree with any of that, although I'd stick a big citation needed on the implicit suggestion that there's a large group of scientists who aren't making a good-faith effort to ensure that their successors will have the information they need to reproduce (that is, after all, what a paper is).

My issue is the flippant and silly claim that "[i]t's possible to imagine a version of academia where results that can be attributed to noise don't get published".

[saalweachter]

I think this is actually something that can be experimentally examined.

Take a sampling of a large number of papers, give them some sort of rating based on whether they provide enough information to reproduce, how clear their experimental and analytical methodology was, whether their primary data and scripts are available, etc, and then look at that rating versus their citations.

Hopefully, better papers get more attention and more citations.

(And yeah, "peer review" as it is done before a paper is published is not supposed to establish a paper as correct, it is supposed to validate it as interesting. Poor peer review ultimately makes a journal uninteresting, which means it might as well not exist.)

[wyattpeak]

That sounds like a very interesting idea. At the least, it would be interesting see the major classes of reproducibility problems. And there may well be a lot of low-hanging fruit, as the comments on this page suggest about data corpuses in computational fields.

[lsh]

> However, real scientists don't stop and accuse the original authors of being "unprincipled" -- nine-point-nine times out of ten, they work with the original authors to discover the discrepancies.

I'm not a real scientist or even a pretend one, and I'd like to believe your 9.9/10 figure, but don't delude yourself there aren't those out there publishing papers for the sake of nothing more than retaining their position in a university. Or bumping their citation count or pushing an agenda or whatever.

We're in this 'reproducibility crisis' precisely because this game of science being played doesn't reward reproducibility and scientists are just as much participants as publishers are.

[BenoitEssiambre]

The statistics and probabilistic methods of science are there specifically for controlling and quantifying the effect of noise and uncertainty and making sure experiments are reproducible a high percentage of the time.

If you don't get a quantifiable amount of reproducibility, there is no point to using statistics at all and what you are doing is not science.