[keldaris]

I don't know that we can reform - if you spend too much time trying, the system will automatically cull you. Personally, I'm lucky enough to be at least slightly insulated from the true scale of the problem by working in theoretical/computational soft matter physics, where the costs/grants/impact factors are comparatively on the small end of things. Medicine and biology seem to be affected the most, for good reason - this is where the messiest problems intersect with the most interest from society at large, so you get the most acutely misaligned incentives.

That being said, communicating limits of applicability and degrees of certainty to a popular audience is hellishly difficult even if you're trying to be perfectly honest. Even in a hypothetical world where we've somehow fixed academia, this will always be a hard problem most scientists are ill equipped to tackle.

[staticautomatic]

I bet a lot would change if the largest funding institutions publicly prioritized funding scientists who have a history of publishing reproducible work.

[keldaris]

Sure - for some definition of "reproducible". Any metric you come up with that's easily measurable will be gamed into oblivion. Anything meaningful - like looking at dedicated reproduction studies, etc. - requires a complete change in how academia functions, because right now publishing a paper that just aims to reproduce (or fail to) a previous work is effectively impossible in most fields. Not to mention that given the incentive structure you'd be pretty crazy to even try.

[staticautomatic]

It may well be some kind of law that every positive incentive creates at least one perverse incentive, but that doesn't mean positive incentives don't work or aren't worth trying.

[lettergram]

I tend to argue that 10%-20% of a grant should go to reproducibility. This does two things:

1. It would help fund / ensure funding to several labs in the space.

2. It would help ensure reproducibility of results

Yes, it can be gamed, but generally it should be easier to reproduce results so 10%-20% of the funding to follow directions should be okay. Of course, this could lead to one group just constantly doctoring results to show something is not reproducible. In which case, a third lab would need to get some funding to check it out.

[Fomite]

First they'd have to fund the reproducibility studies.

[taneq]

Well they shouldn’t have any trouble funding the first one...

[Fomite]

They have plenty of trouble funding much of anything.

[cgiles]

Completely agreed on all counts.

By "reform from within", I mainly meant the NIH, NSF, big funders, etc, need to pay more than lip service to this before Congress gets involved. Although there are people who have built careers on irreproducibility itself. Ioannidis for example. But that requires a lot of dedication and tends to piss a lot of people off.

Another counterintuitive possibility is that scientific publication could go more of the PLoS route and actually lower standards for initial publication -- and then rely more heavily on post-publication peer review. Then irreproducible papers would get publicly "downvoted". And conversely, papers that didn't seem useful enough at the time to make it into Nature, but turned out to be pathbreaking, would get the recognition they deserve.

Further incentives for publishing as much raw data as possible and, where applicable, code, would help too. The NIH has done a good job here. They require a lot of high-throughput datasets to be made available raw if they were collected with NIH funding. They provide hosting for this. It means people can go back and re-analyze it, and you don't have to trust the authors' analyses.