[placidpanda]

The main reason that I think I make good decisions in the absence of data, is because of how much I've relied on whatever data is available to inform my decisions and learn going forward. This is a huge advantage to multivariate testing as a practice/culture. As a consequence, it's often very easy for me to pick out when readouts are giving a deceptive answer (i.e. oh, the scope of this uplift is too much, we need to double check if something happened to negatively impact the control).

I'm not sure I'd agree that people are often operating "close enough to optimal", but I would definitely agree that integrating experimentation is hard enough that sometimes the effort (or mistakes) you can introduce will cause more problems than you're helping. But I think this is more a function of how poor people are at the mechanics and the mindset of running experiments than that they're doing good enough pricing hot dogs. Experiments in many places are looked at for either CYA or boasting about quarterly results and not to truly learn/grow/improve.