[westiseast]

sorry, yes you're right. The sample size was not huge (@500 people viewed each variant) although the result was statistically significant. I just added a note to the post about this. My aim is to repeat the test at a later date to see if the results repeat themselves.

[marekmroz]

Correct me if I am wrong, but isn't statistical significance a function of sample size?

[cmurphycode]

Yes-- sample size, and effect size. He's saying that the results were strong enough that given the sample size, they were statistically significant.

Simple example: if I have a study of 20,000 people and I find that 51% of people prefer foos to bars, that may be similarly significant to your study of 20 people, all of which prefer foos to bars.

[VMG]

You are right.

The author should plug the numbers in here: http://www.usereffect.com/split-test-calculator

I'd go as far as to consider the data to be worthless without the confidence value

[westiseast]

If I've understood it correctly, the smaller your sample size, the bigger the difference you need between response rates. So if the difference is 60%, then a sample of 30 people can be statistically significant. But if you're looking for a 5% improvement on something, you'll need to sample a large number of people.

I just used this:

http://www.prconline.com/education/tools/statsignificance/in...