[di456]

> Part of the reason why many companies jumped on this bandwagon is also because 'customizability' is hard(er) to build in, and certainly more expensive to maintain.

It's also harder to A-B test with so many variables. If tests aren't statistics significant, the value of user analytics and UX experimentation decreases from a "% lift" perspective. It's harder to know if a feature change or a user defined config had a causal relationship to some other metric.

It may be the A-B testing tail wagging the dog.

[nvarsj]

That's a good insight. A bit like unit tests, A-B tests drive the design of the website to make them testable. Not sure that is ultimately the best choice for UX, but here we are. I suppose it makes it a lot easier to justify one's perf rating.

[di456]

Maybe on the perf rating, but on a bigger scale it could make it way more complex for analytics departments to function. Complexity adds real costs when that department exists to increase revenue and retention, and iterate quickly.

Those types of tests serve two very different purposes. UI is also unit testable.

Unit tests are more of a binary pass/fail. A-B tests are looking for cause-and-effect relationships by comparing some metric between a control group and a variant group.