[davidkuennen]

I connect data-driven to A/B testing. And A/B testing for me is like squeezing the maximum out of an orange, instead of making more oranges.

It will give a short term boost, but eventually you will have to look beyond your data and explore new things your customers really want.

[Enginerrrd]

I think this is akin to focusing too hard on things that are appear easy to measure over things that are (perhaps incorrectly) perceived as hard to measure.

The parable of the drunk found looking for his keys under the light instead of inside the dark bar where he lost them comes to mind. What's needed is a flashlight, however imperfect.

[jlei523]

A/B testing can definitely make product teams feel like they're making progress but it should not be a team's only strategy.

It's a bit like moving the furniture around the same room.

[cestith]

Nobody has a big enough test set to A/B test every combination of elements for a web page, a product feature set, or even a conference presentation. The combinatorics are just unwieldy.

Someone has to have multiple good ideas and the ability to carry them out properly before an A/B test is even valuable. Otherwise you're measuring one uninformed random change against another. A/B confirms whether you've succeeded in improving something. It can't really suggest what to try next.