[marcosdumay]

I'm not really disagreeing, but the 5-sigma rule is there because the hypothesis is not formulated before you run the experiment.

If you make the hypothesis first, 3-sigma is quite enough. Many physics experiments do exactly that, but famous high-energy ones don't.

(That said, not having an hypothesis beforehand was very common in psychology before the 21st century.)

[kjkjadksj]

This is one of those rules of thumb that don’t make any sense to me as someone who works with data in the field. You conduct the same exact experiment with the same conditions and same data and get the same result. But whether you speculate on a hypothesis before or after suddenly changes the significance threshold with no actual change in the underlying data or method? Did you cast some ancient spell when you came up with the hypothesis or something?

[marcosdumay]

> with no actual change in the underlying data or method

Proving a known hypothesis or deciding what you want to know after the fact are completely different methods.

> Did you cast some ancient spell when you came up with the hypothesis or something?

"I'll know it when I see it" is an incredibly vague way of doing science that requires extra rigor somewhere else to compensate.

Or, in a maybe better explanation, testing for multiple hypothesis is subject to this kind of failure:

https://xkcd.com/882/

So you need more data confirming your theory.

But if you state your single hypothesis beforehand, you are at the situation of the top square, with a high-confidence result.