[warwick]

Asking if design makes a difference is kind of a silly question. We know it makes a difference. Using A/B testing, or even just by changing one element at a time, we can measure improvements in things like conversion ratios or download rates.

Of course, to measure if each part of your design is effective, you have to change it in isolation. I wouldn't surprised if there are elements of the new Wordpress site that decrease conversion, as well as some which are increasing it. If the design had been iterated on, instead of just rolling out the new design, we might know which elements should have been changed to create better results.

[sgdesign]

A/B tests show their limits when it comes to design. You can't A/B test a brand or a logo, or even a background or a font. And you can't test elements of a new design in isolation, that doesn't make any sense. Elements of a design have to work together as a whole.

[moxiemk1]

Everything you've listed except for a "brand" (and the usage here seems pretty ill-defined) is very A/B testable. I'm not really sure how to refute that, but the idea that they aren't is ridiculous. Would that be effective and produce good results? Maybe not - that's a great discussion to have, but whether or not it's possible isn't.

Similarly, you're right - elements of a design do not exist in isolation. However, any design will have a series of choices that resulted in its various elements. Some choices are small/minor enough that there exist other elements that also work together with the whole, and so A/B testing them isn't some insult to the idea of design, but an at least plausible decision-making tactic.

[sgdesign]

Well ok, you could A/B test replacing Arial by Helvetica, but it would be pretty meaningless and probably not make any difference. That doesn't mean that there is no difference. Maybe on a subconscious level, people would start to perceive your site and relate to your brand differently.

And maybe changing the font, and then the background separately would not make a difference, but changing both at the same time would make an impact. All those things are A/B testable in theory, but I don't think it would be very effective.

[warwick]

If you think the background and the font together are going to make a change, then by all means, conduct a test and see if whichever metric you're working with goes up. I'm in favour of isolating changes as much as possible, but if you believe that two elements of your design work only in concert with each other, then they should be modified accordingly. These aren't random permutations you test, they're new tweaks to a design that you suspect might help.

Sure, changing the font may be meaningless, but at least with a test, you KNOW it's meaningless. I try to let experience and taste guide my changes, but without checking that the change is effective you're just guessing.

Design exists to further a goal. Sometimes the goal is brand related, which I'll admit is just about impossible to measure, and sometimes it's seeing if you can reduce the frequency of support requests. One of these is obviously a better candidate for A/B testing.

[Edited to fix a spelling error]