[AJ007]

#1 Its good to see a post using statistically significant sample sizes. Most articles on this topic that I have read either include sample sizes so embarrassingly tiny it raises questions if the author even understood his statistics class (30 to 300) to simply not including numbers at all.

#2 If your audience is already pre-sold on your product when they visit your site, landing page changes won't be as meaningful. Possibly relevant to Jitbit, in this case.

#3 When you do conversion optimization, eventually you hit a number that simply becomes unbeatable. Stating the obvious, you can't convert at over 100%. Depending on traffic source/quality/intention you may find that ceiling to be lower, around 60-70%.

[btilly]

In reverse order.

#3. For real companies with completely unmotivated visitors, 60% conversion rates from visitor to paying customer is untenable. 5% is probably too much to hope for.

#2. Absolutely. A/B tests should be focused on actual balanced decision points. "Should I sign up?" "Should I open this email?" "Should I go back to this site?" But not on people who are already committed to doing what they were going to do anyways.

#3. On sample sizes, it is important to think carefully about the maximum effort you're willing to put into a test. Be very, very cautious about accepting test results that arrive early. No, 99% confidence is not enough to stop with 200 conversions, and 99.9% probably isn't either. Don't worry about statistical significance when you kill tests that have run too long to be worth continuing. This is a line of reasoning that is sadly rare in our industry. (I keep meaning to write my next article on that topic. But http://elem.com/~btilly/ab-testing-multiple-looks/part2-limi... explains one way to come up with such a strategy.)

[pbiggar]

Looks at OP's point about traffic source. Its likely that ones of your traffic sources converts way better than 5%, while unqualified people straight from google will convert much lower.

[m4xt3r]

#2 is totaly relevant to our Macro Recorder app.

Macro Recorder is really popular and most of the visitors are already "pre-sold". They don't care about the landing page.

We used to run tests on that page, because it has 1k daily visitors, we thought that we could test ideas more quickly. That was before we realised, that we can't rely on the results.

We even used false conclusions, based on those results, to make changes to landing pages for other products. Perhaps the biggest mistake we've made.