Finally! It looks like someone actually did some simple math to determine if their results were statistically meaningful. I wish more people would include a section like this when they talk about conversion rates.
Running a one-sided Student’s t-test with a means difference of 14%, our experimental data yields a P value of 0.000051. This means that there is only a 0.0051% chance that we would obtain a 14% (or greater) improvement if the real effect wasn’t at least this large.
Err... Student's t doesn't assign a probability to P(measured>=14 | actual<14), it assigns one to P(measured>=14 | actual=14) under the assumption of equal variances (which here means actual=0, so clearly not applicable).
I figure that the actual effect is 16.05% +- 0.17%. And that's simply by assuming 145k is close enough to infinity, and for IE.html only.
That is, unless I'm mistaken about this, since I grok only the fairly basic stuff. Which is certainly plausible in and of itself.
Running a one-sided Student’s t-test with a means difference of 14%, our experimental data yields a P value of 0.000051. This means that there is only a 0.0051% chance that we would obtain a 14% (or greater) improvement if the real effect wasn’t at least this large.
Err... Student's t doesn't assign a probability to P(measured>=14 | actual<14), it assigns one to P(measured>=14 | actual=14) under the assumption of equal variances (which here means actual=0, so clearly not applicable).
I figure that the actual effect is 16.05% +- 0.17%. And that's simply by assuming 145k is close enough to infinity, and for IE.html only.
That is, unless I'm mistaken about this, since I grok only the fairly basic stuff. Which is certainly plausible in and of itself.