[tribeofone]

"No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it."

This is not entirely true, if you have enough traffic, even the slightest change can make a difference. HOWEVER, for 99% of sites what you say is true. A/B testing should be used last, to squeeze out the last few percents of optimization, once you've done everything else. Not as a 'first step'.

One cool thing that can come of these tests, as an anecdote, is to see what is really working on a page, as opposed to what needs to be optimized.

We ran a Multivariate Test once for a signup form. The results were all similar, except for 1 which was much, much worse then the others. We could not understand why because there were no big differences between the elements, but this one particular combination set was having a bad time at it. Turns out, the headline was longer for one of the variations. This wrapped the text and pushed the call to action down below the fold, resulting in a ~30% (negative) difference in the conversion from the ones that were above the fold.

[m4xt3r]

Those things happened (and still happen) a lot to us. At first we were all excited when there was a significant difference between alternatives, but now I just assume that something is wrong by default.