Hacker News new | ask | show | jobs
by majormajor 1927 days ago
That doesn't sound right. The reason sample size is important is because the larger the group, the less likely the group is to be atypical. I don't think you can turn that around and then say "this group is small, which itself is evidence, because otherwise it would have to be extremely atypical to show a difference." That's completely circular.

The p-value is still the important thing because the p-value is itself affected by the sample size.

2 comments

No, effect size very much informs all of that.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/

> the larger the group, the less likely the group is to be atypical.

Not true (unless I misunderstood what you were trying too say). The probability that a sample will be atypical under the null is exactly the significance level, and does not depend on its size.

What does change with size is that the distribution around the statistic gets more concentrated, meaning your threshold moves to the left, towards lower effect sizes.

Which also means that for smaller sample sizes, it is harder to reach significance, unless you're dealing with a fairly large effect size.