Hacker News new | ask | show | jobs
by gamedori 2502 days ago
When I see a headline like this my instinct is to jump to "overfitting".

In a study like this they use P=0.05 as their cut off for significance. One in 20 such studies would be expected to give meaningless results. But they had the choice of cohort (all kids or boys and girls separately), so 1 in 10. They measured urine concentrations of fluorine at three time points during pregnancy, which makes for 7 different potential combinations of input variables, for a total of 14 different potential analyses, where 1 in 20 is expected to be significant. Of course, they didn't preregister, so who knows whether their analysis changed on the fly..

1 comments

Good points. It's even more than 1 in 20 when you realize that studies does no get published with p > 0.05, so in the "published studies" sample this effect should be even larger.