[vobios]

> we see PhD students and post-doc working 70h hours week on experiments with seemingly random results until the randomness goes their way.

There are known and tested protocols that can fail. Not every step can be accurately recorded. It's very common that an experiment will not work well the first time it's performed (even when supervised by someone experienced). Over time, researchers improve their skills and achieve better results following the same exact protocol. Does that mean that the science behind the experiment is bad?

[nkurz]

Does that mean that the science behind the experiment is bad?

No, the science might be solid. But if attempts by peers to reproduce the results fail more often than they succeed, the paper describing the science is (by definition?) inadequate. The level of detail required in the paper varies from field to field, and experiment to experiment, but if the techniques aren't described well enough for others to follow them, then the paper needs more detail.

[vobios]

Even commercial products (where they have a financial incentive to provide clear and comprehensive instructions) often take a lot of training before they work properly. How can we expect a small research lab to be better?

[nkurz]

You are right to point out that giving clear instructions for a complex task is difficult. And elsewhere in this thread, 'nycticorax' makes some great points. I fear my answer is along the lines of Rutherford's often ridiculed quote about statistics and experimental design: "If you need to use statistics, then you should design a better experiment."

If the experiment that you describe in your paper is too difficult for others to reproduce, perhaps it shouldn't yet be published as a paper? Would the public interest be better served by funding researchers who do simpler but reproducible work, rather than complex work where the results essentially need to be taken on faith? Carried to the extreme, this is a terrible rule, but I feel there is a kernel of truth to it.

I guess the right strategy depends on how much faith you have in the correctness of published results, evaluated solely on plausibility and the reputation of the researcher, and thus how much value there is in a conclusion based on irreproducible results. I think there is a currently a justified crisis of belief in science, and that many fields would do well to get back on to solid ground.

But it's a wicked problem.