[hinkley]

> So let's circle back to metrics and ways to ensure we use them for guidance rather than obey them reflexively.

I'm not sure this is the right framing of the problem. "Guidance" is too nebulous and leaves us with the same problem of interpretation.

    Advice from an old tracker: You want to find someone, use your eyes.

When I have someone who is getting themselves/us in trouble by looking at charts, my remedy always includes the notion that charts are for asking questions, not answering them. You might have a chart that says user response time has improved, but have you tried using the service from your phone (with wifi disabled)? If the chart says the times are shifting, we need to know why they are shifting, because that's information. "Looks like response time went down," is only useful as the introduction to a question, like, "did we change something?" or "did we have a routing issue?", or "can we do the same thing on this other service that is drawing complaints?"

Getting good questions also often involves cleaning up sources of noise in the charts. It's too easy for someone with an axe to grind to jump to conclusions that this is a repeat of a problem we had before (ie, this is Team X's fault... again) or that lack of improvement means failure to act. It is not impossible that I fix one regression at the same time someone else causes another, or changes the denominator by the same magnitude as the fix.