[sgdesign]

Well ok, you could A/B test replacing Arial by Helvetica, but it would be pretty meaningless and probably not make any difference. That doesn't mean that there is no difference. Maybe on a subconscious level, people would start to perceive your site and relate to your brand differently.

And maybe changing the font, and then the background separately would not make a difference, but changing both at the same time would make an impact. All those things are A/B testable in theory, but I don't think it would be very effective.

[warwick]

If you think the background and the font together are going to make a change, then by all means, conduct a test and see if whichever metric you're working with goes up. I'm in favour of isolating changes as much as possible, but if you believe that two elements of your design work only in concert with each other, then they should be modified accordingly. These aren't random permutations you test, they're new tweaks to a design that you suspect might help.

Sure, changing the font may be meaningless, but at least with a test, you KNOW it's meaningless. I try to let experience and taste guide my changes, but without checking that the change is effective you're just guessing.

Design exists to further a goal. Sometimes the goal is brand related, which I'll admit is just about impossible to measure, and sometimes it's seeing if you can reduce the frequency of support requests. One of these is obviously a better candidate for A/B testing.

[Edited to fix a spelling error]