[Slix]

> No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it.

This contradicts the article about A/B testing by Wired, which used the Obama campaign as an example.

http://www.wired.com/business/2012/04/ff_abtesting

[RyanZAG]

Couple points there:

#1 The article is very vague - the Obama team may have actually made a very common mistake and stopped the tests early when they hit a significant level (they were in a big rush at the time, they had days to get everything working, not months). We don't know anything about samples sizes, etc.

#2 The main detail they talk about changing was changing from a movie to a still picture. This is a lot different than changing a button's color - many people will see a movie and close the page if they are in a quiet environment such as work or a library.

So it could very well contradict the A/B testing article by Wired and still be correct - or it could be wrong. Contradiction itself is not a red flag here in something as new as A/B web testing.

[m4xt3r]

Alright, that was a bold statement on my side. That's true for 99.9% of the companies, not 100%.

Again, most of us don't have enough traffic to make successful buton color tests.

[justinmarsan]

Have you tried MAB ? It seems like the best way to do A/B testing on small-traffic pages, let it run forever without losing much due to the reduced frequency of poorly performing variantes.

Setup your test with 3 or 4 colors, the programm will automatically pick the best one over time, so it doesn't matter how long you let it run, be it weeks months or years, and when you don't want to test it anymore you pick the one that performed the best out of the many visitors you've had over a large period.

http://en.wikipedia.org/wiki/Multi-armed_bandit

[mjhea0]

thank you!

i cannot tell you how excited i get when a client wants to split only to change a color of a button or a subtle change in a particular font. so frustrating!