[gipp]

When the author says:

> However, that assumes that someone presenting an analytical presentation will be viewed more favourably than someone presenting something softer. Basically, I had assumed a data-driven culture exists, when in reality businesses are struggling to create that culture in the first place.

I think this understanding of the situation is in itself part of the problem. It assumes that someone coming in with an analytical presentation necessarily should be viewed more favorably than someone presenting something softer.

Coming from someone who's been working as a data scientist for several years, data-driven decision making has its limits. One very important one is a strong, strong bias towards myopic metrics (e.g. "engagement" over "lifetime value", "traffic volume" over "reputation in the market"), on the basis that they:

* Are more easily measurable

* Provide more data to work with

* Provide a stronger signal/noise ratio

* Provide much faster feedback

An organization which _always_ values data-driven decision making over expertise-driven decision making is always going to fall prey to this myopia. Fighting cargo-cult data science and building a sustainable analytical culture also means understanding the limits of data-driven decision making and that it does not replace, but supplements, "softer" expertise-driven culture.

[Bartweiss]

> An organization which _always_ values data-driven decision making over expertise-driven decision making is always going to fall prey to this myopia.

This is a huge and under-appreciated concern. It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach.

A goal like "improve clickthrough rates" is easy to measure, but without some human insight it's all too easy to achieve it at the cost of overall success. Did you decrease time-on-page? Maybe your visitors feel mislead. Did you decrease conversion rate? Maybe your new visitors don't actually want your product. And so on, indefinitely, including lots of side effects you might not have convenient statistics for.

I have a depressing sensation that at least half of corporate data science consists of abusing Goodhart's Law - finding a useful metric and then naively optimizing for it until it's no longer representative of business success.

[Barrin92]

>It's disturbing how often success is measured by optimizing a single metric, and how resistant people can be to recognizing issues with this approach.

this is very true. The zeal of data driven approaches sometimes reminds of "craniometry" where people tried to gauge intelligence by measuring the shape of one's skull.

The trade off of using quantitative methods is always that you might lose too much meaning. The good thing about data driven approaches is that they are transparent and enable objective decision making, but people need to pay close attention and be alert that whatever it is they are measuring still has some qualitative justification.

[riri-au]

You preempted my followup article. I'm not sure where the balance between these two is, but I'm sure that many places get it wrong.

[Bartweiss]

I'm definitely excited for that followup, then. I think this is an incredibly common issue in both directions, and a surprising number of companies seem to make both errors at once.

[dmix]

This is an interesting point.

I'm curious if one of the 'myths' of a data driven company is that you can instantly begin making decisions fed by 'real-time data' and learn after-the-fact from feedback loops. But for many legacy businesses the data pipeline for their important KPIs still moves slowly.

And then the data that does come in quickly becomes over-valued because everyone was sold the idea of instant gratification. So there is pressure to react to things quickly like meaningless web traffic metrics or local sales data - which may fluctuate heavily on a daily basis - instead of waiting for relevant patterns to emerge over longer periods.

Statistical significance and error rates are then overlooked in the name of a cargo cult data culture.

This is why business books can be dangerous or even destructive, as business advice from one person's experience is sold as generic design patterns that apply to every business - which isn't the case. This is why understanding the business inside-and-out is the most important attribute, then having MBA-esque skills/toolset is useful second. So you take the reality of the business into full consideration and apply tools to it, rather than seek out tools and pigeonhole your business into them.

[conjectures]

On that point, Superforecasting by Tetlock is an excellent book on softer analysis, which will make perfect sense to quant readers.

[gipp]

Oh, not familiar, looks good! Ordered.

[riri-au]

That's a really good point, and I've seen that myopia cause problems as well. Like you say, it's about understanding the limits and applicability of data science, and how it interacts with experience and qualitative strategy.