[dsugarman]

this is all fine and good, but if you're goal is to see what works best between X new versions of a page and you are rigorous in creating variants, Optimizely is a great tool for figuring out the best converting variant.

[pdpi]

Except, apparently, they aren't actually that good at _that_. If an A/A test to not yield 100% chance of 18% uplift, what gives you any degree of certainty that other tests won't have equally skewed results?

[vitamen]

Run an A/A/B (or A/A/B/B) test, decide on traffic levels before you start the test, and let it run until you reach those levels before you peek.