|
It seems like I see these articles pop up on a regular basis over at Inbound or GrowthHackers. I think the problem is two-sided: one on the part of the tester and one on the part of the tools. The tools "statistically significant" winners MUST be taken with a grain of salt. On the user side, you simply cannot trust the tools. To avoid these pitfalls, I'd recommend a few key things. One, know your conversion rates. If you're new to a site and don't know patterns, run A/A tests, run small A/B tests, dig into your analytics. Before you run a serious A/B test, you'd better know historical conversion rates and recent conversion rates. If you know your variances, it's even better, but you could probably heuristically understand your rate fluctuations just by looking at analytics and doing A/A test. Two, run your tests for long after you get a "winning" result. Three, have the traffic. If you don't have enough traffic, your ability to run A/B tests is greatly reduced and you become more prone to making mistakes because you're probably an ambitious person and want to keep making improvements! The nice thing here is that if you don't have enough traffic to run tests, you're probably better off doing other stuff anyway. On the tools side (and I speak from using VWO, not Optimizely, so things could be different), but VWO tags are on all my pages. VWO knows what my goals are. Even if I'm not running active tests on pages, why can't they collect data anyway and get a better idea of what my typical conversion rates are? That way, that data can be included and considered before they tell me I have a "winner". Maybe this is nitpicky, but I keep seeing people who are actively involved in A/B testing write articles like this, and I have to think the tools could do a better job in not steering intermediate-level users down the wrong path, let alone novice users. |