Hacker News new | ask | show | jobs
by 9mit3t2m9h9a 3114 days ago
5% false discovery rate is true if the apriori probability of each result is 50%. If a journal wants to publish only surprising results, and accepts p=0.05 with a good methodology as true, it will have higher rate of false claims, because the most interesting things are more surprising than a coin toss.

And if you slice the data from a single experiment in 40 independent ways, your chances to get something with purely random significance p<0.05 are better than 50% for a single study…

1 comments

It doesn't "mean" that the a priori is 50%, I guess what you want to say is that if we consider the a priori probabilities are equal between the two hypothesis, then we can use the 5% p-value.

But if the probability of the hypothesis is much less likely, we might need a p-value much lower to be sure.

Edit: my comment doesn't mean much since you edited your first sentence.