[weland]

> That's both untrue and fallacious (see: no true Scotsman).

No it's not. A sound scientific approach requires that a theory be based on reproducible results. If an experiment that verifies your theory confirms your result today and infirms it tomorrow, then the theory, the experimental approach, or both, are wrong.

Of course, experiments that can't be reproduced are part of the scientific endeavour. Every discovery comes at the end of a long sequence of experiments with results scattered all over the grah. But treating them as anything other than stumbling steps that help you refine your understanding of the problem or as dead ends is as unscientific as it gets.

[omginternets]

>But treating them as anything other than stumbling steps that help you refine your understanding of the problem or as dead ends is as unscientific as it gets.

Which isn't at all what I'm suggesting.

I'm arguing against the idea that applying the scientific method and getting a false positive makes the effort unscientific.

So yes, it is both untrue and fallacious.

[weland]

> Only a quarter of scientific drug research is successfully reproduced as well.[1]

The article you are mentioning in [1] refers to the fact that a quarter of the published drug research is not successfully reproduced.

That doesn't mean that people published papers saying "Hey, we did this experiment. Its results cannot be consistently reproduced, so we think it's unconclusive/because our theory is flawed with regards to this or that/because the experiment was flawed with regards to this or that and we think it can be refined by changing this approach or that apparatus".

It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X", but it turns out that their experiments cannot be consistently reproduced, so they're proof of exactly nothing.

That is unscientific.

[omginternets]

>It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X"

This is patently false. Publication is never a claim of conclusive proof; it's a claim of evidence.

I'm sorry, but you are wrong about this. False-positives don't suddenly make the experiment un-scientific. You're very misinformed about how science works:

- False positives are part of the landscape

- Contradictory evidence is part of the landscape

- The above issues are resolved by tracking reproducibility of results

You can come to a wrong conclusion using valid scientific means. The scientific method hinges on the assumption that research will eventually converge on a correct result.

[weland]

Where did I say publication is a claim of conclusive proof?

I said they published papers in which they claimed they conclusively proved something, and it turned out they didn't conclusively prove anything. Specifically, because their results couldn't be reproduced.

In case you're not familiar with how experiments are carried out in natural sciences, "results couldn't be reproduced" means that

1. They claimed they got <these results> with p < <this threshold>

2. Some other guys repeated the same experiment ("repeated" as in they administered the same substances, to a sample of equal size under similar conditions and measured the same parameters under similar conditions) and it turned out that on their results, p was through the roof.

In some cases, that was simply because the authors didn't publish enough information for their experiments to be repeated (I was close to making that mistake, too. Thank God for review committees). But in most cases, that simply happened because authors cherry-picked data or "optimistically" interpreted results.

(Edit: Responsible review committees can sometimes spot the latter, but it's very hard to deal with the former. The correct thing to do is to have all researchers publish all their experimental data, even the one which wasn't included in the papers. A lot of researchers agree, but you'll find that a lot of companies that employ researchers actively invent reasons why their researchers shouldn't do that.)

> If you set your p-value threshold at .05, then one in twenty experiments will produce a false positive.

> The reason is simple: given a p-threshold of .05, one in five experiments will yield a false positive.

Make up your mind already.

[omginternets]

>Where did I say publication is a claim of conclusive proof?

Exactly where you typed It means that a quarter of the published papers say "Hey, we did this experiment which offers conclusive proof of X"

Again, this is patently false because they did not publish papers claiming conclusive proof. They published papers claiming evidence in favor of a theory.

>Make up your mind already.

There's no reason to be disrespectful over a mistake. I meant 1 in 20 (5%, hence the mix-up).

Returning to the point, it takes incredible mental gymnastics to argue that a false positive automatically degrades the status of a study from "scientific" to "unscientific":

1. The adjective "scientific" describes a method, not a result. Those speaking of "scientific results" are either (a) referring to "results of a scientific study" or (b) confused about what science is (namely: a method, not a result).

2. A false positive degrades the status of a result (not a study) from "evidence in favor of X" to "not evidence in favor of X".

I must respectfully insist that you are wrong.

[weland]

> Returning to the point, it takes incredible mental gymnastics to argue that a false positive automatically degrades the status of a study from "scientific" to "unscientific"

No one said anything about A false positive!

"Cannot be reproduced" means there were a lot of false positives. So many, in fact, that you can't really draw any conclusion from the experiment. (Edit:) Or more to the point, that the p value the original authors claimed was bullshit.

Reproducing an experiment means reproducing both the experimental technique and the sample.