[cocoflunchy]

    When A/B testing, you need to always remember three things:

    The smaller your change is, the more data you need to be sure 
    that the conclusion you have reached is statistically significant.

Is that a mathematically provable result? It seems hard to conceptualize what a 'small' or 'big' change is. I would have expected another argument along the lines of "If you make more than one change at a time, you are not going to be able to know which one of your changes caused the result".

[ska]

This property is quite intuitive. Small and big here are relative to the variance of the underlying distributions.

Simple case: think about trying to decide if as normal distribution has mean 0 or mean 1. If the std dev is 0.001, it won't take you very many samples to be fairly confident to this resolution, but of the deviation is 1000, you'll need a lot of samples.

Similarly if the std deviation is only 1, but you are trying to decide if the mean is 0 or 0.001, Far more samples needed.

The intuition generalizes quite well. In the OP case, typically requires sample size estimates will be proportional to the square of the ratio between the size you want to measure and a deviation estimate.

[noloqy]

His choice of words is not very fortunate. If webpage A has an underlying conversion ratio of X, and webpage B has an underlying conversion ratio of Y, then if X is not equal to Y, in theory we could always find evidence that X is "significantly" different from Y.

If X is close to Y, we need a very large sample to achieve statistical significance on the hypothesis that they're not equal, whereas if X and Y are far apart, it is likely that a small sample will already indicate that fact.