[baron_harkonnen]

I once mentioned Goodhart's Law to a data scientist at a company and they immediately rejected it based on the unironic assertion that

"that would mean that KPIs shouldn't be the sole measure of our performance that that doesn't make sense!"

My experience in the field has been that an astounding number of products have been destroyed and users harmed by failing to heed Goodhart's Law.

[dhosek]

Are you sure it was unironic? Because I sure can't imagine anyone saying that unironically.

[marcosdumay]

I've heard plenty of "what else do I have to work with?", that has about the same meaning.

[rhizome]

The politician's syllogism comes to mind: "Something must be done, this is something, so we must do this."

https://en.wikipedia.org/wiki/Politician%27s_syllogism

[gipp]

That at least acknowledges that it's an unsatisfactory situation, OP's conversation didn't even have that level of awareness.

[dboreham]

Presumably someone, somewhere, thinks KPIs are a good idea.

Edit: but that person is unlikely to be subject to KPIs themselves.

[AnimalMuppet]

Re your edit: Not necessarily. They may be a winner under the KPI regime, and may feel that they are less likely to be so under a saner regime.

For example, if I'm a manager that can make my people hit their KPIs, and my KPIs are about getting my people to hit theirs, then I'm subject to KPIs, and I like it. It's easier than making my people succeed at what really matters, and it makes me look good.

[hpoe]

To push against this point I prefer KPIs, or something objective that I can be measured against, now that doesn't mean I like bad KPIs but the fact of the matter is there are always going to be KPIs the only question is how explicit or implicit they are.

When KPIs are explicit everyone knows what they are and can modify their behavior to optimize for their KPIs when all measurement goes away the new KPI is the arbitrary one held in the decision makers head, and now instead of it being an explicit bar that can be objectively used to make decisions the entire system falls apart into politics and emphasizing appearances over work because the only thing that matters with implicit KPIs are what everyone else thinks of you, which is much easier to manipulate than the amount of cash you brought in.

[bonniemuffin]

I regularly find myself saying "the only thing worse than metrics is not having metrics."

[dang]

Is Goodhart's Law a sort of upper bound on the usefulness of data for decision-making in general?

[baron_harkonnen]

Rather than implying a limit to the usefulness of data, I find it speaks more to the folly of substituting data for critical thinking.

You can reduce a fever by treating the underlying infection, soaking in ice water, or taking acetaminophen. No good doctor would judge a patient's health solely because a single metric, namely body temperature, was able to be moved into acceptable ranges. That doesn't mean temperature isn't extremely valuable data, and essential to decision making, but that it cannot be a substitute for understanding and solving the real problem.

I once knew of a SaaS company that had perpetually growing MRR (Monthly Recurring Revenue), great right? Except, churn was also growing. An increase in MRR was achieved by upselling a perpetually shrinking group of core customers. The core KPI of this company was MRR, and, unsurprisingly, this company does not exist anymore. Again, here is a case where we can see all that other data (churn, upselling) is very useful, as is the KPI. But the key to success or failure here is whether or not you want to really expend the effort to understand the problem or just chase a KPI.

KPIs are seductive because they make managing team's performance seem much easier: just get this number higher and you're doing good, get it lower and you're doing bad. But that's like playing a game of chess where each piece is controlled by a different person, and that person is judged solely on how many times they can get the king in check.

[cratermoon]

> No good doctor would judge a patient's health solely because a single metric, namely body temperature

That's a good example because as Strathern's formulation notes, the problem lies in make the metric the target. It would be folly to think that reducing a patient's body temperature to the normal range is sufficient for curing illness. GE's Jack Welch famously focused solely on the stock performance as a measure of success. It worked, by that measure GE was wildly successful. By almost any other measure Welch destroyed GE https://www.bnnbloomberg.ca/jack-welch-inflicted-great-damag...

[some_furry]

I wonder if measuring managers' understanding of Goodhart's Law would result in better management.

/s

[iaw]

That's horrifying, my litmus test for data scientists is whether they acknowledge KPI's are imperfect measures. In my experience, data scientists that come from hard science backgrounds (e.g. Physics and Biostats) tend to be much more open minded to the idea that there isn't a "perfect" statistical measure.

[disgruntledphd2]

Relative to what background exactly?

Like I'd agree that maths/cs people are little naive about error but I can't imagine any data scientist from a social science background thinking that any measure is perfect.

[iaw]

Apologies, I rarely see data scientists from social science backgrounds.

[disgruntledphd2]

Really?

Wow, so when I worked at a FAANG, I would say that soc sci people (broadly defined) made up at least a third of the data science org. Data science is a broad church though, and varies a bunch across companies so it's normal I supppose (though strange to me).