[weland]

> Only a quarter of scientific drug research is successfully reproduced as well.[1]

This is a debate in semantics: research that is not successfully reproduced is not scientific.

It should be part of that which society frowns down on as non-verifiable testimony.

Natural sciences or engineering-related research that cannot be reproduced is nothing but scientific theater. There are various reasons why people do that (including financial ones in the drug industry), but that's a discussion for another time.

This is not proof that "scientific rationality" is brainwashing people. If anything, it's proof that non-reproducible research dressed as legitimate science is dangerous -- so dangerous, in fact, that it can put lives in danger (e.g. when it happens in drug research).

[omginternets]

>This is a debate in semantics: research that is not successfully reproduced is not scientific.

That's both untrue and fallacious (see: no true Scotsman).

If you set your p-value threshold at .05, then one in twenty experiments will produce a false positive. As such, plenty of research is conducted in a benevolent and meticulous fashion, only to yield a non-reproducible result. It's still scientific; it's just not true.

I don't agree with subliminalzen, but (with all due respect -- really!) your comment is hogwash.

[weland]

> That's both untrue and fallacious (see: no true Scotsman).

No it's not. A sound scientific approach requires that a theory be based on reproducible results. If an experiment that verifies your theory confirms your result today and infirms it tomorrow, then the theory, the experimental approach, or both, are wrong.

Of course, experiments that can't be reproduced are part of the scientific endeavour. Every discovery comes at the end of a long sequence of experiments with results scattered all over the grah. But treating them as anything other than stumbling steps that help you refine your understanding of the problem or as dead ends is as unscientific as it gets.

[omginternets]

>But treating them as anything other than stumbling steps that help you refine your understanding of the problem or as dead ends is as unscientific as it gets.

Which isn't at all what I'm suggesting.

I'm arguing against the idea that applying the scientific method and getting a false positive makes the effort unscientific.

So yes, it is both untrue and fallacious.

[weland]

> Only a quarter of scientific drug research is successfully reproduced as well.[1]

The article you are mentioning in [1] refers to the fact that a quarter of the published drug research is not successfully reproduced.

That doesn't mean that people published papers saying "Hey, we did this experiment. Its results cannot be consistently reproduced, so we think it's unconclusive/because our theory is flawed with regards to this or that/because the experiment was flawed with regards to this or that and we think it can be refined by changing this approach or that apparatus".

It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X", but it turns out that their experiments cannot be consistently reproduced, so they're proof of exactly nothing.

That is unscientific.

[omginternets]

>It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X"

This is patently false. Publication is never a claim of conclusive proof; it's a claim of evidence.

I'm sorry, but you are wrong about this. False-positives don't suddenly make the experiment un-scientific. You're very misinformed about how science works:

- False positives are part of the landscape

- Contradictory evidence is part of the landscape

- The above issues are resolved by tracking reproducibility of results

You can come to a wrong conclusion using valid scientific means. The scientific method hinges on the assumption that research will eventually converge on a correct result.

[weland]

Where did I say publication is a claim of conclusive proof?

I said they published papers in which they claimed they conclusively proved something, and it turned out they didn't conclusively prove anything. Specifically, because their results couldn't be reproduced.

In case you're not familiar with how experiments are carried out in natural sciences, "results couldn't be reproduced" means that

1. They claimed they got <these results> with p < <this threshold>

2. Some other guys repeated the same experiment ("repeated" as in they administered the same substances, to a sample of equal size under similar conditions and measured the same parameters under similar conditions) and it turned out that on their results, p was through the roof.

In some cases, that was simply because the authors didn't publish enough information for their experiments to be repeated (I was close to making that mistake, too. Thank God for review committees). But in most cases, that simply happened because authors cherry-picked data or "optimistically" interpreted results.

(Edit: Responsible review committees can sometimes spot the latter, but it's very hard to deal with the former. The correct thing to do is to have all researchers publish all their experimental data, even the one which wasn't included in the papers. A lot of researchers agree, but you'll find that a lot of companies that employ researchers actively invent reasons why their researchers shouldn't do that.)

> If you set your p-value threshold at .05, then one in twenty experiments will produce a false positive.

> The reason is simple: given a p-threshold of .05, one in five experiments will yield a false positive.

Make up your mind already.