Hacker News new | ask | show | jobs
by 2T1Qka0rEiPr 1835 days ago
Yes, I read down to this point and stopped. What significance can you really draw from a sample size of 26? (Serious question to academics out there)
2 comments

To answer your serious question, n=26 is more than sufficient depending on effect size and sampling method.

Think about it, if you have 26 people randomly sampled and you give half a pill and the other half a placebo and the first half dies moments after ingestion, how confident are you that the pill is poisonous?

If the effect size is small, you may fail to detect it with a small population. If the sampling is biased, then you’re going to have a problem even if you have a massive population studied.

The "small sample size" has been drilled to most people as a knee jerk red flag indicating a statistically unreliable conclusion, but this is not always the case. Depending on the desired outcome, a small sample size may actually increase the reliability of the result. This seems like one of those cases.

Think of it like this. The aim is to show a statistically significant difference. With a large enough sample, you will eventually show significance (hence Fisher's famous anecdote of "get more data"), but the effect size may be trivial. Whereas, with a small sample, only non-trivial effect sizes will achieve significance, and therefore achieving significance with such a small sample tells you something about the nontrivial extent of the effect size.

So you could argue that the extent of methylation change was so large, that it achieved significance in a sample even as small as n=13!