[snake_doc]

Obligatory p-value snippet from the ASA:

   P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

Ronald L. Wasserstein & Nicole A. Lazar (2016) The ASA Statement on p-Values: Context, Process, and Purpose, The American Statistician, 70:2, 129-133, DOI: 10.1080/00031305.2016.1154108

[comte7092]

To follow that up (so people know what they actually are), what p-values represent is the likelihood we would observe our data, given the null hypothesis.

Setting a cutoff of .05 is saying “if there’s less than a 5% chance we’d see this data, assuming the null hypothesis, then we can assume that the null hypothesis is false”

[nickff]

But this statement only applies to a 'naive' (or first) statistical analysis or test on the dataset. Once the researcher starts changing their assumptions in response to the results they're seeing, they're p-hacking and p-values are no longer meaningful. In addition, once you have multiple researchers looking at the same dataset with different assumptions, and you factor in publication bias, the p-value also loses meaning.

[comte7092]

Well, yes and no. The p-value still means the same thing, but when you take a dataset and go looking for any result that is under a certain threshold, you’ll probably find it. “Unlikely” events happen all the time!

What your comment is highlighting is an issue with bad experimental design. (And, obviously, with our publication regime)