I guess it is up to you to define your metric to be measured and to be optimized. To define the right metric and targets is not easy and probably is business-specific, i.e., is not the same for every business.
Seriously, +1 to this. Defining key metrics is something where you should sit down with a data scientist that is familiar with your domain. In my experience, the problem isn't that people can't A/B test, they don't know what to test or how to determine if an experiment was "successful".
You are right. Another detail is to find a metric which is meaningful in long term, e.g., the one which is in line with long-term happiness of your customers.