[superposeur]

Yes, the devil always seems to be in the details in psychology experiments. Were the experimenters giving subtle cues to the child, and was this simply a test at how deftly the child picked up these cues? What was the exact wording of the “deal” offered to the child and would a wording change alter the results? Was the experiment conducted at a time of food scarcity or abundance? What were the prevailing cultural norms of how a child “ought” to behave? Would the results change if average child age was 6 months older or younger when experiment was conducted? What was in the drinking water and the air and the paint at the testing site? (With strong claims in the literature that all these are correlated with measures such as average population IQ!)

In the face of all these potential confounders, more statistics and controls seem necessary than, say, in a physics collider experiment on electrons (each electron possessing exactly two characteristics, location and spin, and all such electrons behaving identically regardless of location or time). Yet, even in this setting of simplicity and reproducibility, physicists have still found it necessary to establish a stringent, five-sigma threshold for discovery — 3 sigma anomalies come and go. Such a stringent threshold is unthinkable in psychology due to practical considerations. Ergo, it’s hard for to see how psychology can become a reliable empirical science.

[marcosdumay]

I'm not really disagreeing, but the 5-sigma rule is there because the hypothesis is not formulated before you run the experiment.

If you make the hypothesis first, 3-sigma is quite enough. Many physics experiments do exactly that, but famous high-energy ones don't.

(That said, not having an hypothesis beforehand was very common in psychology before the 21st century.)

[kjkjadksj]

This is one of those rules of thumb that don’t make any sense to me as someone who works with data in the field. You conduct the same exact experiment with the same conditions and same data and get the same result. But whether you speculate on a hypothesis before or after suddenly changes the significance threshold with no actual change in the underlying data or method? Did you cast some ancient spell when you came up with the hypothesis or something?

[marcosdumay]

> with no actual change in the underlying data or method

Proving a known hypothesis or deciding what you want to know after the fact are completely different methods.

> Did you cast some ancient spell when you came up with the hypothesis or something?

"I'll know it when I see it" is an incredibly vague way of doing science that requires extra rigor somewhere else to compensate.

Or, in a maybe better explanation, testing for multiple hypothesis is subject to this kind of failure:

https://xkcd.com/882/

So you need more data confirming your theory.

But if you state your single hypothesis beforehand, you are at the situation of the top square, with a high-confidence result.