[btilly]

It looks like you forgot to rescale the binomial distribution.

If X_i is a series of independent, identically distributed random variables with mean m and variance v, then X_1 + X_2 + ... + X_n is approximately a normal variable with mean nm and variance vn. Therefore

(X_1 + X_2 + ... + X_n - nm)/sqrt(vn)

is approximately a standard normal.

If you draw that graph, visually the two lines should lie right on top of each other. To see the problem you need to zoom in on the tail and blow it up, and only then will you see the issues with the convergence.

[sesqu]

You're right, I did have the wrong scale, and should have realized. I believe this plot to be correct. Black is binomial, red is normal, both for the difference in conversion rates.

http://img267.imageshack.us/img267/6691/binnormdifflikelihoo...

The normal approximation seems reasonable, and so a t-test shouldn't cause problems. I made a plot of the relative error of the normal approximation; the tails are indeed too fat. In particular, there's a bump to the left of 0, so a t-test would slightly overestimate p.

http://img205.imageshack.us/img205/6691/binnormdifflikelihoo...

[btilly]

The problem is that when you're near confidence you're in the tail, and the student t-test is extremely sensitive to the shape of said tail. If n is large enough this difference will be washed away as everything converges to normals. But with smaller sample sizes the difference can be quite significant.