[everdrive]

I don't have a way to prove this, but I've long suspected that A/B testing might be one of the major culprits to the modern problem of software churn. ie, constantly-changing GUIs, features, and products. Some percentage of users will always have problems with an application or interface. Constantly chasing every user hurdle might sound like a good idea, but I do wonder if it's similar to listening to too much fan feedback in the case of video games, movies, etc. Implementing every fan suggestion will usually make a terrible game. Instead, the ideal is to figure out which fan suggestions to ignore, and which to listen to. Only some of them will work for a given application.

I wonder if A/B testing is a bit like that. You're suddenly listening to every single issue encountered by users, but lack the wisdom to understand which issues to ignore. As a result, software changes constantly, users cannot really learn a GUI because it changes so frequently, and software teams feel like they're constantly improving things, however the software itself is not actually getting better when a the whole user's experience is taken into consideration.

[deltathreetwo]

The churn is the result of the fact that we don't understand what is optimal.

We know how to optimize for things like the shortest distance between two points but we don't understand how to optimize Software Design or even GUI.

This is literally what "Design" means. If something needs to be designed it means there's no theory or notion behind it on what it means to be "optimal." So we "design" a "better" solution but we don't actually know if it's better. So we design another solution and the cycle continues.

A/B testing when done on a population that gives consistent answers should converge on a consistent solution. If the population gives different answers at different times then of course there will be churn.

I would say the methodology of A/B testing is indeed like machine learning, it's quite good and accurate. It's the data source that's the problem. If you have users that don't know what they want or behave differently and inconsistently then your conclusions reflect the data. Perhaps the data is accurate and there is no consistent conclusion, OR the data is inaccurate and you need to get it from a better source other then users telling you what they think is better.

[marginalia_nu]

Honestly, this comes off more as a model problem than a data problem.

Here are some dubious assumptions:

* There is an optimal answer.

* Human preferences are transitive (if I prefer A over B and B over C, then I prefer A over C).

* Human preferences are internally consistent.

* Human preferences are stable over time.

Apply these assumptions to finding an answer to the question of what you should make for dinner, and you'll quickly see there are problems.

[webnrrd2k]

> Apply these assumptions to finding an answer to the

> question of what you should make for dinner, and you'll

> quickly see there are problems.

And, to amplify your point a bit, there are problems with A/B testing for which of my children I should love the most, or what I find beautiful or interesting; or should I A/B test out whether enslaving others works out well for me?

I chose extreme examples, but the point is that there are many, many things in human experience that don't lend themselves to easy, simplistic rules. It's genuinly hard to work through a lot of a lot of the issues that people face in real life.

With that said, it's it's also important to work through and to understand objective data as best as possible. For clearly defined and nicely-behaving problems objective data is certainly the way to go. The problem is that a lot of the problems people actually face aren't so easy to understand, and aren't so well-behaved.