[totaldex]

I understand the frustration, but A/B testing is one of the more objectively tools we have at our disposal.

While there are good arguments against A/B testing UI changes and doing p-hacking, much of the modern web's current UX and UI improvements are in part due to this. How else would we truly know what affects user on a broad scale?

[saagarjha]

In theory, yes, A/B testing sounds great. However, whenever I see it implemented it invariably becomes some sort of annealing process for a metric that doesn't actually make the site more pleasant for end users ("engagement", usually) and you have people shipping apps with seven different UIs in them and users who are accustomed to features moving around and disappearing completely unpredictably. As I was searching for a picture of "Courier" Google gave me a completely different "cards" UI that went back to how it was before when I searched "Menlo". My last couple weeks with Slack on iOS have been a nightmare as it constantly switches between its fairly decent UI and some new abomination depending on which workspace I'm in and the phase of the moon. It seems like nobody really knows how to do A/B testing properly. (Perhaps the companies that do are doing it in such a way that I cannot notice. But there is ample evidence of a lot of products where I can.)