[jacquesm]

Surprisingly simple question that is very hard to answer well I think....

If you only test one variable at the time then you simplify your tests to the point where you can extract some metric to determine whether you've improved or not compared to the old situation.

Nothing stops you from then doing more ab testing with other combinations relative to your 'new best'. This may include going back to the original setting with some other variable changed, that way you avoid the local maximum problem.

So say we have a site in position 'A', we make version 'B' and we test them against each other. If we find out according to our chosen metric that 'B' performs better we now have several choices:

We can do another A/B test starting from 'B' changing some value to see if we can improve on 'B' directly, or alternatively we can go back to 'B' versus 'A' + some new modification that is not 'B'.

I hope that's clear....