[kqr]

While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1].

One would think this is cognitively dissonant enough, but it gets worse:

This article, with the thesis that good arguments are more important than data, is based on, well, a good argument – not much data. On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning. (There's some, yes, but the main thrust of why I believe it is that variants of the experiment have been replicated reliably.)

I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time.

----

Edit a few minutes later: This actually prompted me to refresh on the subject. It might be the case that Meehl is actually making the same argument as this article, only it gets distorted when repeated. Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.

----

[1]: Here's just one relatively early example: http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/...

[baryphonic]

Implicit in all of this is the is-ought problem.[0] The data are collected and interpreted under some procedure, often with normative biases built in about how the world ought to be (especially when involving human subjects), but are interpreted as saying what the world is. Thus data collection is fertile ground for charlatans.

When the psychiatric profession or Google or whoever else use experimentation to decide on what criteria they should follow, with sound controls, valid statistical analysis and loads of replication, they either arrive at evaluation procedures without much bias or, more likely, they realize the phenomenon they're trying to measure is almost all noise with no or excessively weak signals.

A better approach would be to acknowledge as much normative bias as possible up front, then conduct tests using sound experimental design. But the problem with this approach is that the data shows performing a bunch of well-crafted experiments is expensive, and management doesn't buy in if the vast majority are unlikely to reject the null. That leaves us which a class of "data driven" managers who are in fact indulging their biases to a sometimes extreme degree, using "the data" as a shield.

[0]https://plato.stanford.edu/entries/hume-moral/#io

[zmgsabst]

I find it strange that these are presented in tension, when they’re complementary.

You can create situations where you have a lot of data but can’t reach conclusions, because you lack a narrative and explanatory model which “makes sense” of that data; inversely, you can convincingly argue complete nonsense that’s obviously contrary to facts.

Deep understanding requires a model/narrative which fits the collection of data we have, and which allows us to reason about and predict the outcome of new situations.

As Jeff Bezos put it:

> Good inventors and designers deeply understand their customer. They spend tremendous energy developing that intuition. They study and understand many anecdotes rather than only the averages you’ll find on surveys. They live with the design.

> I’m not against beta testing or surveys. But you, the product or service owner, must understand the customer, have a vision, and love the offering. Then, beta testing and research can help you find your blind spots. A remarkable customer experience starts with heart, intuition, curiosity, play, guts, taste. You won’t find any of it in a survey.

https://www.aboutamazon.com/news/company-news/2016-letter-to...

[gchamonlive]

I was about to write that in case of Bezos with Amazon, the customer was simpler and the answer was to just pour money into it until you substituted the market, but I realise now that that is not that simple. It seems simple because we have hindsight.

My main idea though is that it is very hard to foresee what the customer will want after you deliver the product. Not what the customers want now, because sometimes they don't understand it until they experience it, and that makes me think that there is a LOT of luck at play here and a good deal of continency in prototype product design. Experience alone could be overrated. Think Kodak, I don't think they didn't have experience in product design, that they didn't understand their customers. I think they only didn't risk their luck and didn't think about what their customers would want in the future. And that is always a gamble.

- Things are more nuanced and complex than I am putting it here, but bottom line is that I am trying to tap into survivors bias.

[zmgsabst]

Sure — business is a gamble, made harder by our own foibles. My main point was that even somewhere very data-driven like Amazon, that data should be used within a narrative as a grounding-not-guiding force.

(Disclaimer: I used to work on a customer sentiment analysis team at Amazon, doing a lot of surveys.)

Amusingly, the two paragraphs after what I cited agree on that danger:

> The outside world can push you into Day 2 if you won’t or can’t embrace powerful trends quickly. If you fight them, you’re probably fighting the future. Embrace them and you have a tailwind.

> These big trends are not that hard to spot (they get talked and written about a lot), but they can be strangely hard for large organizations to embrace. We’re in the middle of an obvious one right now: machine learning and artificial intelligence.

I don’t think the digital revolution was lost on Kodak — I think that for organizational reasons they couldn’t pivot.

> The first actual digital still camera was developed by Eastman Kodak engineer Steven Sasson in 1975. He built a prototype (US patent 4,131,919) from a movie camera lens, a handful of Motorola parts, 16 batteries and some newly invented Fairchild CCD electronic sensors.

https://www.cnet.com/google-amp/news/history-of-digital-came...

[ricardobeat]

Seems far-fetched to assume that this thesis applies to product development just the same?

The impact a data-driven mindset can have on the organization cannot be understated ('RIP intrinsic motivation' section). I've seen it first-hand, both data being used as cop-out for bad leadership, meaningless 'successes' used as trading cards for promotions, and design experts having a decade of experience overridden by shaky statistical analysis, or worse, non-inferiority tests.

Meanwhile, the shortcomings in the product that everyone knows are rarely addressed because they are 'difficult to test'.

[klenwell]

> They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment.

I came across this in Thinking Fast and Slow. Kahneman was a big fan of Meehl and restates the point:

The important conclusion from this research is that an algorithm that is constructed on the back of an envelope is often good enough to compete with an optimally weighted formula, and certainly good enough to outdo expert judgment.

https://www.goodreads.com/quotes/9574537-the-important-concl...

I too agree with the premise of this article. On this topic of expert judgment vs data, however, I found the counterpoint in this HN comment thought-provoking enough to bookmark and refer back to now and again:

I started at MS during Vista and I've been involved (sometimes tangentially) with Windows ever since. This is all my opinion, but It's been very interesting seeing the decision making process change over time.

If I had to summarize the change, I'd say that it's evolved from an expertise-based system to a data based system. The reason why eight people were present at every planning meeting is because their expert opinion was the primary tool used in decision making. In addition to poor decisions, this had two very negative outcomes:

1) reputation was fiercely fought for. Individuals feared that if they were ever incorrect, the damage to their reputation would limit their ability to impact future decisions and eventually lead to career death. Whether this actually happened or not is irrelevant; the fear itself caused overt caution and consensus seeking.

2) In the absence of data, an eloquent negotiator is often able to obtain their desired outcome, no matter how sub-optimal that outcome might be.

https://news.ycombinator.com/item?id=15174737#15176957

Even more provocative, it ends up being a (qualified, as I read it) defense of telemetry.

[int_19h]

It seems to imply that expertise-driven design gave us Vista and Win7 while the data-driven one gave us Win8, Win10, and Win11. It's notable that, from this list, Win7 seems to be the only one that people genuinely liked.

[iamcurious]

Yup, it seems a side effect of data driven approach is that Windows no longer cares about its own reputation.

[einpoklum]

Expertise-driven design did not give us any Windows operating system. I don't believe that MS Windows is the kind of system OS experts would design.

But - perhaps you're referring to the user interface? Or just the kernel? Or the driver mechanism?

[throw_nbvc1234]

Define "people". Tech people, people/customers in general, some other group such as shareholders? Both your point and the point your responding to could be true at the same time both anecdotally and/or in the data. Anecdotes are probably just another form of "expert opinions"

[int_19h]

It's entirely an anecdote, but from my experience, Win7 was broadly accepted as a good iteration among techies and non-techies both.

As a software engineer, I actually find a lot more to be excited about in Win10+ thanks to WSL and other such things. But I don't hear my acquaintances who are non-techies being positive about anything from Win8 on.

[dwaltrip]

> Edit a few minutes later: This actually prompted me to refresh on the subject. It might be the case that Meehl is actually making the same argument as this article, only it gets distorted when repeated. Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.

Highlighting your edit at the bottom, as I think it’s important and not everyone will read that far.

[3pt14159]

I've come to heavily discount these types of studies. What makes an expert? What was the sample size of experts? What was the non-expert tool? Etc.

There is such a thing as having common sense based on thoughtful life experience. Checklists and regressions help, but human beings are very capable of deep expertise and to pretend otherwise is silly. I expect a musician to be able to identify a violin from a viola.

[bumby]

>Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.

Maybe too much of a nit-pick, but how does one build expertise without data? I'll grant that it may be informally or subconsciously collected but it's still data.

It makes me think of Malcolm Gladwell's book Blink. There are lots of experts who can subconsciously chunk data to make intuitive and reliable decisions. But they got to that point often gathering lots of data in the form of experience.

[lo_zamoyski]

> This article, with the thesis that good arguments are more important than data, is based on, well, a good argument – not much data.

I'm not sure what you're claiming. All intellectual demonstration is a matter of rational argument. That's what proofs are: arguments. Data is not self-explanatory or demonstration. "Data" can only support arguments by first being collected, something motivated by argument, and then interpreted so that it can enter into argument as a body of propositions.

> On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning.

I don't understand. Argument is logical demonstration. The strongest form is the deductive argument. If you don't have a logical argument, then you haven't got a demonstration.

> I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time.

Depending on what you mean, this could be good or bad. Inconsistency is not a virtue, and if there is an inconsistency between two of your beliefs, then it means you've got work to do (or at least you'll need to admit you don't know what the truth is). This requires humility, the frank acknowledgment that you're faced with an aporia that you don't know (at least not yet) how to address. It also requires patience if you are to tolerate your ignorance instead of jumping to some ersatz explanation.

[uneoneuno]

I feel like the author is leaning into comfort, intuitiveness. You bring up a fantastic point. Often we find data reveals things very unintuitive to human experience. We should always try to make Good Arguments - but without data they aren't always honest beyond feelings.