[kqr]

The threshold for significance lies in the eye of the beholder. A particle physicist might not be satisfied with anything over 0.01 %. A social scientist might be happy to see 10 %.

The 5 % number you mention is completely arbitrary and often woefully inappropriate.

Look at it from a betting perspective. Can you earn more than 10 × your investment if the null hypothesis is false? Then anything less likely than 10 % is significant.

[robotresearcher]

It's a convention for scientific reporting. Your trades are not bound by this convention.

The parameter value is not arbitrary. It's a convention arrived at after hundreds of years. If it were arbitrary, p=0.999 or p=0.00001 would be just as good. We've settled on p=0.05 being usefully convincing but not crazy demanding to obtain by experiment with noisy measurements.

[aaplok]

Null hypothesis testing was invented less than 100 years ago by Fisher, who completely arbitrarily picked 0.05 [0]. That value was not arrived at through wisdom of experience, and certainly not after hundreds of years of practice.

Though it has now indeed become conventional to test with p=0.05, there is nothing wrong with reporting an effect that fails the null hypothesis test. At least that is the position of the American Statistical Association [1].

[0] https://www.cantorsparadise.com/what-is-the-meaning-of-p-val...

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5017929/

[robotresearcher]

Thanks for these refs. I read [1] carefully and I take your point that it’s ok to strictly report whatever the data says.

On the value itself, we are quibbling about the meaning of ‘arbitrary’: Fisher certainly could have chosen another value, but not all values would be considered useful. Some expertise about the nature of real world data and the minds of statisticians is encoded in the chosen value.

If I propose that we change the convention to use 1e-12 instead and you think ‘that’s too small, I prefer it the way it is’, then it’s not arbitrary in the sense I mean.

[kqr]

The thing you seem to be missing is that there's no one number that's a meaningful limit for all purposes.

What probability you accept as significant should depend entirely on how you plan to use the results. Something with a p value of a staggering 70 % (i.e. it's more likely not true than true) is significant if the payoff is good when it's true, and the cost is small when it's not true.

And 70 % is very far from 5 %!

Then again, if the payoff is tiny compared to the cost, you might ask for a p-value of less than 0.01 %, in order for it to make sense to take the chance on it.

Think like a poker player: a hand that has 1/4 chance of winning needs better than a 3-to-1 payout when it wins to be playable. Conversely, when the pot offers you a 3-to-1 payout, you better make sure your hand has more than a 1/4 chance of winning.

[robotresearcher]

I understand and agree with that. Pascal’s Wager is the extreme example: unlimited upside makes the likelihood irrelevant.