Deriving formulas for the expected sample size needed in A/B tests

Y	Hacker News new \| ask \| show \| jobs

	Deriving formulas for the expected sample size needed in A/B tests (camdp.com)
	27 points by zenburnmyface 4562 days ago

2 comments

jdlshore 4562 days ago

I'm not qualified to comment on Cam's math, but Evan Miller's tools [1] are my go-to source for calculating sample sizes and significance. How does this differ? (If it does.)

http://www.evanmiller.org/ab-testing/sample-size.html

link

yetanotherphd 4562 days ago

That looks right to me. This is a very important calculation to do in order to avoid getting 100x the number of samples that you really need.

Essentially the same calculation applies to classification error.

link

ska 4562 days ago

The real (and all to common to run into ) problem isn't having 100x the samples you really need, it is having 1/10 the samples you really need... And not knowing that.

link