[DanBC]

> I don't think this actually tells you about what your customer's like. But I'm having a hard time saying why. If I'm presented with iTunes but I don't like iTunes and never click through, does that really say anything useful about whether I like the alternatives?

You're performing the test on one large group of people, not on lots of individual people. (Does that make sense?)

It doesn't tell you what Bob as an individual likes, but it might tell you what a collection of people prefer.

It's quite important with A/B testing to set a sample size at the beginning of the experiment and stick to it. A statistician will be able to explain why.

Because you're not testing individual people it'll be tricky to disentangle the reasons for preference from the data.