[lalaland1125]

A quote from the Brian himself illustrating how he is able to "publish" so much junk research.

> P-hacking shouldn’t be confused with deep data dives – with figuring out why our results don’t look as perfect as we want. With field studies, hypotheses usually don’t “come out” on the first data run. But instead of dropping the study, a person contributes more to science by figuring out when the hypo worked and when it didn’t. This is Plan B. Perhaps your hypo worked during lunches but not dinners, or with small groups but not large groups. You don’t change your hypothesis, but you figure out where it worked and where it didn’t.

This is horrifically bad research practice and basically guarantees that all of your results will be completely p-hacked.

https://statmodeling.stat.columbia.edu/2018/09/23/tweeking-b... is another hilarious example.

[ip26]

Really? Suppose you just ran a twenty year, many million dollar field study, the hypothesis was not confirmed, you just throw everything out & that's that?

[lalaland1125]

No, you publish a negative result, which in many ways can be more or just as informative as a positive finding. See for instance https://www.nejm.org/doi/full/10.1056/NEJMsa1906848.

It's totally fair to do subgroup analysis, etc to try to better understand the data, but you have to keep in mind that you can't claim statistical significance on those without very careful control over researcher degrees of freedom and other related issues.

[mattmanser]

Yes because firstly, you didn't design the experiment for that reason, so you didn't consider what inadvertent biases you introduced for the "new" way of looking at the data. Plus on top of that, because of false positives, you're going to find a LOT of false positives by accident.

So every study "discovers" something new, but it's not because of biases and false positives.