[antux]

A/B testing can be useful at times but it's largely overrated because you're only discovering the best design out of the ones you test. That means there could be a far better design that you failed include in the experiment.

Just because one design converts more than the other doesn't mean it's the design with optimal UX. I've seen many tests where the designs included already had faulty UX. This is why it's better to have a trained UX designer on your team who can fix basic flaws and present the best version of various designs for testing.

[rwilson4]

I agree, but that's not the whole story. A/B testing isn't just for identifying the best option, it's for figuring out how much opportunity there is in a particular facet of your business. If you try a few reasonable designs, and some designs have much better performance than others, then it makes sense to continue investing time to further improve. You can take what you've learned and try to come up with even better designs.

On the other hand, if the variants mostly perform the same, why spend more time on it? Go focus elsewhere.

It certainly is a logical possibility that the next design you try will be much more impactful, but after trying several variants unsuccessfully your time is probably better spent elsewhere.

[antux]

That's my point with having a trained UX designer who knows what they're doing. They can improve a design much better than the average developer or designer because they have more training. An average designer or developer is going to hit a ceiling and be spinning their wheels creating variants with minimal improvement. This is why a/b testing is overrated. In other words, too much emphasis placed on testing, testing, testing and not enough on UX design.

[ssharp]

I'm not entirely sure what the point here is. You're calling A/B testing overrated because it doesn't involve UX designers? I'd agree that having some UX resources in test design is critical, so if that's being done up to standard, is testing still overrated?

[hinkley]

It’s another hill climbing algorithm, with all of the same problems.