[Leander_B]

Can't really agree much on this article and I have the sense it is somehow written out of frustration and lack of experience.

The '6 hidden costs' can all be addressed, some are not even hidden like time, running tests take time and resources like anything else. The other factors like speed/performance and confusion can all be solved very easily.

Some of my take-aways for testing

- Don't focus on small changes like buttons, copy etc.. these are mostly useless yes

- Make radical changes, e.g. on stickermule can try to have the whole product to make a sticker to checkout on 1 page or start with configuration first and then select sizes etc..

- To filter out false positives, compare same sample of data (e.g. organic traffic is good) for the same period range before a winning test went live and after

Personally I have seen some great results with the above tips in mind, from B2B making visitors perform small actions first or showing a demo vs. presenting them with a form immediately (4x more leads) to e-commerce sites adding a direct checkout vs. normal checkout path (lower avg. order but much higher conversion rate to largely make up for it) etc..

You could argue these are part of product (which a conversion department should be part of, not marketing) but without doing the testing you can only guess about the outcome. And sure many tests are fixing usability (e.g. seen 250% increase in registrations by fixing telephone number prefix and date of birth field formatting) and UI issues but that's part of it.

[ac132]

It's unfortunate to dismiss another view point as you're frustrated or inexperienced while ignoring the actual points made.

The biggest point being #1: Resources are finite. Time spent on A/B testing cannot be spent elsewhere. How you use your time determines your growth trajectory. A/B testing is a growth tactic, but it's not always the best use of your resources.

There is absolutely no way to solve this problem. You'll always have a trade off to make when it comes to how you spend your resources and time. You shouldn't neglect that cost when deciding if a/b testing is necessary for your organization.

Btw, keep in mind, I admit we still do some a/b testing. It's just not an key component of our growth strategy.

[Leander_B]

I agree it might not always be the best use of resources, this depends on a wide array of factors within your business.

What I mean is that don't see it anywhere as a hidden cost in the true sense of that word as you know it takes resources and time, but more so about wrong priorities and business value.

As for the other points, they are so easy to solve:

>>Lost conversions

Trade off you need to make beforehand and need to be aware of with the end goal in mind which is to improve your main conversion metrics.

>>Performance

Most decent conversion platforms use fallbacks and also your way of setting up experiments makes a big difference.

>>Confusion

Letting other departments know what you're doing and what's running, might be harder in big corps though.

>>Speed

This goes for everything in a company I think (new feature/product releases, quick fix vs. proper solution etc.). I personally don't mind spending x amount of time to improve y % in conversion rate, so comes down to setting good KPIs and testing what really is important to your business.

>>Focus

Can be done with a proper conversion team that has all these roles covered (bit hard to set up but possible) or making the conversion department a client within the company. Also the teams mentioned work for other departments as well and improving conversion is a win across the board not just for the conversion team.