[travisjungroth]

The impact of sample size completely defies human intuition. I give approximately zero weight to any opinion about a sample size that doesn’t come with the formula, except for some coworkers talking about specific topics where they’ve run the numbers so many times.

The problem is the sample size relationship with power is at a square, then it involves the effect size and the variance. Quadratic relationships are unintuitive, many order of magnitude differences are unintuitive, and variance is unintuitive. So it’s like a superformula for the human brain to not guess accurately.

I write experimentation software, and with samples in the millions, data scientists still want more power. Then you run some internal experiment with n=20 and it’s like “oh yeah, super significant”.