[jrockway]

I think A/B testing requires A or B to actually be good. Imagine a data-driven restaurant that uses A/B testing to determine what customers want to eat. A is cockroaches. B is tarantulas. The data says that more customers prefer tarantulas! But they still go out of business because the steak next door is much better than either option.

[cthalupa]

>The data says that more customers prefer tarantulas! But they still go out of business because the steak next door is much better than either option.

This is where your analogy sort of falls apart. Amazon seems to be doing the opposite of going out of business to the steakhouse next door.

You can of course get good results out of a bad process, but this is usually not something that happens in a sustained manner over such a long period of time. Processes that result in positive effects for periods of years or decades are generally sound.