[konstantinua00]

every time I hear about examples of simpson in peactice, I don't get what lesson to learn

marketting team overoptimized, so non-nutrition demand fell?

drop nutrition from line of products, so that you're both efficient in products you do and overall?

these metrics are insufficient and it's better to look at gross change rather than ratios?

I have no idea

[brabel]

The article suggests an answer to your question, see the last sentence of the introduction:

"its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once."

In the case above, they failed at "keeping the parts in mind" as clearly, the different ratios between different products was crucial.

[gen220]

IME, the "problem" (to the extent there is one) is almost always that the naïvely-chosen KPI metric wasn't specific enough.

Here's a recent example from a friend. You're a SaaS company, and your home page's load time is reported as slow. You set your KPI for the quarter to be "reduce p99 load time of the home page by 50%".

The load time is a function of customer size, so bigger customers = slower home page. It's actually a quadratic function. So the p99 of small customers is like the p50 of large customers. You have 20 small customers and 20 big customers.

That quarter, the sales team onboards 10 new tiny customers, and 10 big customers churn. It's the holiday season in your big customers' geo, so mostly small customers are using the platform. It's the busiest time of year for the small customers, so they're over-using the platform.

All these factors lead to p99 latency dropping by 60%, smashing the KPI goal. Bonuses all around, pats on the back. And no code changes needed, besides!

The solution is: choose a KPI that is tightly coupled to your problem, and not confounded with other variables.

In the above case, a better KPI would have been "p99 latency for large customers", because it is robust to the distribution of customer sizes across current users, churned users, and seasonal differences in usage.

[thih9]

I think the last one is closest. I’d go with: “finance team should look at the gross change”, if that’s what matters for them.

[infogulch]

Maybe the lesson is to analyze different business units (product categories?) independently first, then the whole.