[squarecog]

co-author of blog post here.

We covered some of this, at a high level, here: https://blog.twitter.com/2015/the-what-and-why-of-product-ex...

Certainly not everything needs to be A/B tested -- depending on the nature of the change and the kind of insight one is looking to get, MAB may be better, if it applies, or a number of other approaches can be more beneficial, including not running an experiment at all. After all, Twitter itself came from a side project at the podcasting company Odeo. If Twitter was a/b tested, I'm sure it wouldn't move any podcasting metrics in a meaningful way :).

I'm obviously biased -- but I've seen really important insights gained from A/B tests, and wouldn't dismiss them too easily.