[simonh]

I think it's important to bear in mind that even with the system working perfectly well, and perfect ethics, we should expect to see a lot of papers published with false results.

Lets say there are 200 propositions we want to test and are candidates for publication, that 20 of them are true and that our error rate is 5%. That means when we test the 20 that are true 19 of them will be accurately shown to be correct and 1 will be erroneously found false.

However when the other 180 propositions are tested 5% of them will be erroneously found to be true, that's 9 propositions. This means we will end up with 28 'successful' studies that make it into prestigious journals, about 1/3 of which are false positives.

And as I said, that's if the system works perfectly with no fraud whatsoever. Throw in some human error and it's not surprising if a fair few studies start to look pretty dodgy. Add in some genuine fraud too and you've got a full-on replication crisis with all the trimmings.