[motohagiography]

Good analysis and great of them to share their thinking. Does feel like this could have been a tweet that said the necessary condition for successful ML solution is applying it to a problem that has asymmetric upside.

Great for telling people they should get tested for diseases, terrible for diagnosis. In the alerting first case, consequences of being wrong are no better than base rate as they wouldn't have been tested otherwise, and the upside saves a life. In the latter diagnosis case, the consequences of being wrong are catastrophic, and it is substituting for the best available judgment. Similarly, it's great for fraud detection, terrible for making credit decisions, because the false negative rate is essentially externalized. It's good for finding opportunities, bad for providing services. So funnels and conversion pipelines it's great for.

So perhaps there's an ironic Turing test for ML solutions that is related to the relationship between the size of a group of people and the effect of mean reversion of their collective intelligence on their behaviour makes them indifferent to the perceived intelligence of the model, whereas a given individual will find the results of the model unsatisfying. From an indifference perspective, AI can fool some of the people all the time, and all the people some of the time, but no confusion matrix satisfies all the people all the time. Economically, ML will be useful for creating simple and cheap services that people who can't afford better will use, and substitute up from them when they can afford better, known as "inferior goods." There may be a hard limit on ML providing "normal goods," to individuals at scale for this reason. Lots of money to be made, but lots to be wasted tweaking your ROC curve to in the hope of creating a normal good.

I yell from the rooftops every chance I get that "the confusion matrix is the product." That is, your FP/FN/TP/TN rate is your product, and you are optimizing your system for the weights your customer assigns to those variables.

There is another ML/DL use case I'm hacking on that is about enabling privacy, but even this reduces to the asymmetry of the upside/downside of the confusion matrix. Obviously the article is more nuanced than this, but I think this heuristic is a key tool for reading articles like it.

[jmatthews]

I appreciate the thoughtful commentary. I couldn't disagree more with you more of course.

There are 2 instances where AI breaks the mold you've cast.

Executing rote tasks that no humans need do, and relatedly, while there does seem to be a tough hurdle when it comes to "better than human" execution there is also an inverted survivors bias. Once a technology is production ready it is no longer AI. Cars aren't robots, antilock brakes aren't AI, Once a system outperforms a human it's technology, not intelligence.

[motohagiography]

Our disagreement might be subtle. An old saw of mine is that the Turing test thought experiment is covered by prior art in economics, where the idea of an indifference curve describes the points between amounts of things where people are indifferent to substituting between them.

I agree these things you state aren't intelligent, but nor are computers, or can they be - people just become indifferent to whether we are dealing with a human or a computer.

My assertion is that we are highly sensitive to substitutes when the downside risk is large, but largely indifferent to them and even like them when they resemble a lottery with good upside at low cost or risk.

Self driving cars are a good example, where someone asked me whether, if I had kids, would I send one to school in traffic in an autonomous vehicle. I told them it would depend on how many kids I had.

But this pretty much describes the dynamic.

[nl]

Self driving cars are a good example, where someone asked me whether, if I had kids, would I send one to school in traffic in an autonomous vehicle. I told them it would depend on how many kids I had.

Pretty sure that answer is much less convincing than you think.

Infact, I thought you were right until that, then realised that was an answer no parent would ever give which made me realise there's a lot missing in your hypothesis.