[m12k]

I think most people overlook/don't know that the needed sample size depends not just on the confidence you want, but also on how big the effect you want to measure is. E.g. if landing page A has a conversion rate of 50% and B has one of 55%, that's going to take a lot of sampling to prove. But if A has 40% and B has 80%, then that's going to show up in the samples very quickly. But exactly which question you ask affects the needed sample size greatly - e.g. showing that 'B performs better than A' will take fewer samples than 'B performs at least 20% better than A'. This makes it much harder to have a correct intuition about needed samples sizes.