[deltathreetwo]

The churn is the result of the fact that we don't understand what is optimal.

We know how to optimize for things like the shortest distance between two points but we don't understand how to optimize Software Design or even GUI.

This is literally what "Design" means. If something needs to be designed it means there's no theory or notion behind it on what it means to be "optimal." So we "design" a "better" solution but we don't actually know if it's better. So we design another solution and the cycle continues.

A/B testing when done on a population that gives consistent answers should converge on a consistent solution. If the population gives different answers at different times then of course there will be churn.

I would say the methodology of A/B testing is indeed like machine learning, it's quite good and accurate. It's the data source that's the problem. If you have users that don't know what they want or behave differently and inconsistently then your conclusions reflect the data. Perhaps the data is accurate and there is no consistent conclusion, OR the data is inaccurate and you need to get it from a better source other then users telling you what they think is better.

[marginalia_nu]

Honestly, this comes off more as a model problem than a data problem.

Here are some dubious assumptions:

* There is an optimal answer.

* Human preferences are transitive (if I prefer A over B and B over C, then I prefer A over C).

* Human preferences are internally consistent.

* Human preferences are stable over time.

Apply these assumptions to finding an answer to the question of what you should make for dinner, and you'll quickly see there are problems.

[webnrrd2k]

> Apply these assumptions to finding an answer to the

> question of what you should make for dinner, and you'll

> quickly see there are problems.

And, to amplify your point a bit, there are problems with A/B testing for which of my children I should love the most, or what I find beautiful or interesting; or should I A/B test out whether enslaving others works out well for me?

I chose extreme examples, but the point is that there are many, many things in human experience that don't lend themselves to easy, simplistic rules. It's genuinly hard to work through a lot of a lot of the issues that people face in real life.

With that said, it's it's also important to work through and to understand objective data as best as possible. For clearly defined and nicely-behaving problems objective data is certainly the way to go. The problem is that a lot of the problems people actually face aren't so easy to understand, and aren't so well-behaved.