[ltjohnson]

You're right, reading the paper that way, they are trying to detect a change in convergence rate from 5% to 5.25%, I was confused by the use of % in two separate contexts in the same sentence. That being said, I think this is not a good argument against A/B testing.

Fair enough, it would take a very large sample (122K is close enough) to detect a change from 5% to 5.25%. Being concerned about a change that small seems really silly unless 0.0025 * N visitors * revenue per user is a big enough number to be concerned with. I contend it won't be unless either

(1) N visitors is very large or

(2) revenue per user is very large.

If (1) is true, then testing on 122K users is not a big deal. If (2) is true you probably want to have a much more targeted approach, like someone doing sales.

[gjm11]

Fourteen relative changes of 5% will double your conversion rate. Or halve it. The cost of an A/B test is small enough that if it took, say, a sample size of 1000 then it would be well worth A/B testing changes that might make a 5% relative difference. On the other hand, if it takes a sample of 122k then indeed you might well decide not to bother -- e.g., because it might be impossible. Which is why "it takes 122k rather than 1k to tell with any confidence" is interesting.

(Rough numbers: suppose you get 1000 visitors per day and convert at 5%, and suppose each conversion is worth $10 to you. Then you're bringing in about $180k/year from them, and a relative change of 5% in that is about $9k. Seems worth doing a modestly-sized A/B test for, but if it takes 4 months then you might reasonably decide to spend your effort elsewhere. (Or, of course, not: the actual cost of doing the test is rather small. But a lot can change over 4 months.)