[teraflop]

A/B testing is great when the value you're trying to optimize can be easily measured, such as short-term conversion rate. If your goal is "users should think the site is aesthetically pleasing", that's going to be a hard one to monitor.

[ryandrake]

But if you're running a business, that's not necessarily going to be your end goal. You want users to "think the site is aesthetically pleasing" because those pleased users [convert more / are more engaged / view more ads], whatever your key metric is. And those are typically easy to measure.

[anandgrafiti]

There are far more variables here to measure. First of all what constitutes that fifty you are thinking of rolling out? Lets say you arrive at it and what's next? For an engaged user the AB testing can measure up as you can define clear set of tasks and desired outcomes. But as i stated above, good type often goes invisible. People engage with the content and the not the,medium to tell you anything tangible. One cannot take away the subjectivity from the process when there is not anything tangible to measure.