[gjm11]

Fourteen relative changes of 5% will double your conversion rate. Or halve it. The cost of an A/B test is small enough that if it took, say, a sample size of 1000 then it would be well worth A/B testing changes that might make a 5% relative difference. On the other hand, if it takes a sample of 122k then indeed you might well decide not to bother -- e.g., because it might be impossible. Which is why "it takes 122k rather than 1k to tell with any confidence" is interesting.

(Rough numbers: suppose you get 1000 visitors per day and convert at 5%, and suppose each conversion is worth $10 to you. Then you're bringing in about $180k/year from them, and a relative change of 5% in that is about $9k. Seems worth doing a modestly-sized A/B test for, but if it takes 4 months then you might reasonably decide to spend your effort elsewhere. (Or, of course, not: the actual cost of doing the test is rather small. But a lot can change over 4 months.)