[dmvdoug]

This is the important and troubling point. Everyone trumpets science as a model of a rational, self-correcting social enterprise. But we see time and time again that it takes non-scientists to blow the whistle and call foul and gin up enough outside attention before something gets done to make the correction. That puts the lie to the notion of self-correction.

[moab]

This is an issue at the department politics level. For the scientific field, once someone starts retracting papers (and arguably, even before this), everybody knows that you should take person X's papers with a huge grain of salt.

E.g., in math / theory, if someone has a history of making big blunders that invalidate their results, you will be very hesitant to accept results from a new paper they put on arXiv until your community has vetted the result.

So yes, I do trumpet science as a model of a rational, self-correcting social enterprise, at least in CS.

Other sciences like biology and psychology have some way to go.

[VHRanger]

The thing is that replication is inherently easy in CS. Especially now that people are expected to post code online.

Forcing authors to share raw data and code in all papers would already be a start. I don't know why top impact factor papers don't do this already.

[moab]

I completely agree. It's a pity that this isn't becoming standard in fields affected by the replication crisis. I would be happy to be corrected if someone has heard / experienced otherwise.

[chasd00]

> you will be very hesitant to accept results from a new paper they put on arXiv until your community has vetted the result.

Forgive my ignorance but I thought that was SOP for all papers. Is it not?

[moab]

Well not really, right? Let's suppose some well known, well respected author that has a history of correct results puts up a new paper. I (and I think most people) will assume that the result is correct. We start to apply more doubt once the claimed result is a solution to a longstanding open problem, or importantly, if the researcher has a spotty track record for correctness (in math/TCS) or falsifying results (in experimental fields).

But really we shouldn't be talking about math errors and falsification in the same category.