[marbletimes]

I agree that statisticians would better than CS people appreciate the importance of uncertainty intervals--it is mostly cultural--but that "In reality there are very few ML applications that don't need confidence estimation and estimation of monetary costs" is empirically false.

If ML application require uncertainty attached to point estimate, we would see plenty more uncertainty intervals attached to point estimates, but in industry, outside of niches (e.g., banking, bio, actuary to name a few), very few bother dealing with them.

I am currently part of a large team (we are talking hundreds) of ML specialists, and I have yet to see a single presentation in which a point estimate was associate with some uncertainty interval. And in my previous company it was the same and when I interview candidates (dozens? hundreds?) I never get a satisfactory answer to the confidence interval vs predictive interval question I ask about.

[antman]

Let me rephrase your empirical observation in probabilistic terms. If the a random sample of data scientists from startups had the same distribution of mathematicians and CS people than a ramdom sample of data scientists from banking then we could compare empirically whether confidence intervals are equally useful in both industries.

Given that historically regulators figured out that when playing with other people's assets you need to assess your confidence, the volatility of the outcomes in non banking industries that lack such oversight can be greatly attributed to people DunningKrugering after a couple of Andrew NG's courses.

That is my claim and based on my experience working in projects accross many industries accross many countries.