[mark_story]

One of the dangers of A/B testing that the author didn't discuss is actually measuring statistical validity. Simply split testing with a low sample size or low level of difference between versions could just be random chance. I find it is always important to figure out the statistical significance of your results to ensure its not just the roll of the dice.

[itsbanderson]

This is a good point, and something we definitely consider. Our A/B testing tool, ABBA, performs the necessary statistics and is mentioned in the "Test everything" section of the post.

[thejosh]

ABBA is a fantastic name for a A/B testing tool. Props to the creators for that name.

[itsbanderson]

ABBA: http://www.thumbtack.com/labs/abba

[adrianhoward]

One of the dangers of A/B testing that the author didn't discuss is actually measuring statistical validity

Isn't that actually one of the problems of thinking you're doing A/B testing when you're actually not ;-) A proper analysis is an intrinsic part of what A/B testing is.

It's like doing TDD without the refactoring step. TDD without refactoring is, well, not TDD.

[jiggy2011]

What is a good way to determine when your sample size is big enough to to warrant A/B testing?

[itsbanderson]

Here's a sample size calculator tool: http://www.cardinalpath.com/tools/sample-size-calculator-too...

And here's a post that addresses this issue: http://37signals.com/svn/posts/3004-ab-testing-tech-note-det...

[harlanlewis]

https://www.google.com/webhp?ie=UTF-8#hl=en&output=searc...