| I've written the same sentence before! This is so cool! pardon the wall of text. Here's my thesis, curious to hear your thoughts. At some time around 2005, when efficient persistence and computation became cheap enough that any old f500-corp could afford to endlessly collect data forever, something happened. Before 2005, if a company needed to make a big corporate decision, there was some data involved in making the decision, but it was obviously riddled with imperfections and aggregations biases. Before 2005, executives needed to be seasoned by experience, to develop this thing we call "Good Judgement", that allows them to make productive inferences from a paucity of data. The Corporate Hierarchy was a contest for who could make the best inferences. Post-2005, data collection is ubiquitous. Individuals and companies realized that you don't need to pay people with experience any more, you can simply collect better data, and outsource decision-making to interpretations of this data. The corporate hierarchy now is all about how can gather the "best" data, where "best" means grow the money pile by X% this quarter. "Good Judgement" used to be expected from the CEO, down to at least 1-3 levels of middle management above the front-line people. Now, it appears (to me) to be mostly a feature of the C-Suite and Boards, and it's disappeared elsewhere. Long-term, high-performing companies seem to have a more diffused sense of good judgement. But these are rare. maybe they always have been? Anyways, as we agree, this has a tendency to lead in problematic directions. Here's my thesis on "why". Fundamentally, any "data" is reductive of human experience. It's like a photograph that captures a picture by excluding the rest of the world. Few people seem to understand this analogy, because they think photographs are the ultimate record of an event. Lawyers understand this analogy. With the right framing, angle, lighting (and of course, with photoshop), you can make a photograph tell any story you want. It's the same issue with data, arguably worse since we don't have a set of standard statistics. We have no GAAP-equivalent for data science (yet?). Our predecessors understood that data was unreliable, and compensated for this fact by selecting for "Good Judgement". The modern mega-corps demonstrate that we don't have a good understanding of this today, evidenced by religious "data-driven" doctrine, as you describe. People will say "hey! at least some data is better than no data!", to which I'll say data is useless and even harmful in lieu of capable interpreters. In 2021, have an abundance of data, but a paucity of people who are capable of critical interpretation thereof. I don't know if it's a worse situation than we had 20 years ago. But it's definitely a different situation, that requires a new approach. I think people are taking notice of it, so I'm hopeful. |