[Bartweiss]

> An organization which _always_ values data-driven decision making over expertise-driven decision making is always going to fall prey to this myopia.

This is a huge and under-appreciated concern. It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach.

A goal like "improve clickthrough rates" is easy to measure, but without some human insight it's all too easy to achieve it at the cost of overall success. Did you decrease time-on-page? Maybe your visitors feel mislead. Did you decrease conversion rate? Maybe your new visitors don't actually want your product. And so on, indefinitely, including lots of side effects you might not have convenient statistics for.

I have a depressing sensation that at least half of corporate data science consists of abusing Goodhart's Law - finding a useful metric and then naively optimizing for it until it's no longer representative of business success.

[Barrin92]

>It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach.

this is very true. The zeal of data driven approaches sometimes reminds of "craniometry" where people tried to gauge intelligence by measuring the shape of one's skull.

The trade off of using quantitative methods is always that you might lose too much meaning. The good thing about data driven approaches is that they are transparent and enable objective decision making, but people need to pay close attention and be alert that whatever it is they are measuring still has some qualitative justification.

[riri-au]

You preempted my followup article. I'm not sure where the balance between these two is, but I'm sure that many places get it wrong.

[Bartweiss]

I'm definitely excited for that followup, then. I think this is an incredibly common issue in both directions, and a surprising number of companies seem to make both errors at once.