[burningChrome]

>> that the only way to provide dispute resolution and customer service to 1B people with only 100 employees is by depriving them of any chance to interact with a human.

Real world evidence supporting your argument:

United Health Group is currently embroiled in a class action lawsuit pertaining to using AI to auto-deny health care claims and procedures:

The plaintiffs are members who were denied benefit coverage. They claim in the lawsuit that the use of AI to evaluate claims for post-acute care resulted in denials, which in turn led to worsening health for the patients and in some cases resulted in death.

They said the AI program developed by UnitedHealth subsidiary naviHealth, nH Predict, would sometimes supersede physician judgement, and has a 90% error rate, meaning nine of 10 appealed denials were ultimately reversed.

https://www.healthcarefinancenews.com/news/class-action-laws...

[thenewwazoo]

> 90% error rate, meaning nine of 10 appealed denials were ultimately reversed.

This is a fantastic illustration of selection bias. It stands to reason that truly-unjustified (some hidden variable) denials would be appealed at a higher rate and therefore the true value is something less than 90%.

That's not to say UHG are without blame, I just thought this was really interesting.

[tkluck]

Your scientific take is useful in the case where selection bias is unavoidable and needs to be corrected for.

This case is not like that; if the insurance agency wants to dispute the 90% false denial rate, it would be trivial for them to take a random sample of _all_ cases, go through the appeal process for those, and publish the resulting number without selection bias.

As long as that doesn't happen, the most logical conclusion for us outside observers is: the number is probably not so much lower than 90% that it makes a difference.

[roenxi]

The insurance company may well have already done that; this is being put by someone who is suing them and looking for reasons that the AI bot is bad. The article is silent on what the company response to the accusation was and, realistically, we'd expect the appealed denials to have a very high rate of error whether determined by bots or humans. Few people indeed are going to waste time arguing a hopeless case against an insurance company - this is classic selection bias.

[sokoloff]

What do you think the claim approval rate is? Less than 10%?

It stands to reason that the overwhelming majority of cases where the claim was approved were approved correctly. Unless that rate is well under 15%, it’s impossible to have the claimed “90% error rate”.

[tkluck]

It's clear from the quoted paragraph that by "error rate" they actually meant "false denial rate". That's also the words I used in the comment you are replying to.

Did you comment because you take issue with misuse of the term error rate, or because you think that correct approvals make up for incorrect denials, and that therefore overall error rate is a useful metric?

[potato3732842]

It's not necessarily even selection bias. It might just be how they roll.

I'd say half of all municipal permitting offices and 90% of welfare/disability offices operate on the principal of "deny them all and see which ones come back around". Wouldn't surprise me at all if the behavior spread to healthcare.

[blitzar]

> has a 90% error rate, meaning nine of 10 appealed denials were ultimately reversed.

feature, not bug

working as intended, closing ticket

[apetrov]

tested, works for me

[sergius]

Not to mention the creation of a single point of failure for a critical service...

[polynomial]

Right, but in this case the critical service isn't providing "health" for users, it's extracting profit from them (from the transactions) for the shareholders. THAT'S the critical service this company cybernetically fulfills.

[tbrownaw]

Seems to me that the use of AI is irrelevant[1], and the real problem is the absurd error rate.

[1] In the sense of "it doesn't matter if it caused the problem", rather than "it probably didn't have any effect". Because after all, "to err is human, but to really foul things up takes a computer".