[rvz]

I have been saying this for months for deep learning in general (and now the new hype in LLMs) in high risk situations such as medical, legal and financial advice and even transportation. The only common use-case which makes sense is summarization and even then, a human expert ends up reviewing the output before post it anyway.

> There are plenty of medical topics which people find embarrassing, and would prefer to - at least initially - talk to a chatbot than to their own doctor.

I don't think you would trust an AI chatbot alone to take a number of pills for any medication instead of going to a human doctor, especially when these AI models risk hallucinating terrible advice and its output is unexplainable and as transparent as a black-box. The same goes for 'full self-driving'.

I don't think one would trust these deep learning-based AI systems in very high risk situations unless they are highly transparent and can thoroughly explain themselves rather than regurgitate back what it has been trained on already.

It is like trusting a AI to pilot a Boeing 737 Max with zero human pilots on board end-to-end. No one would board a plane that has an black-box AI piloting it. (Autopilot is not the same thing)

[MrPatan]

People take pills from known criminals, with high risk of fentanyl OD, just for fun.

Yes, I think people would indeed take pills prescribed by AI, just make it a robot wearing a lab coat.

Also pilots! I mean, pilots kill themselves and a planeload of people more than you think. Of course people would take black box ai that works.

[rvz]

The main difference is humans can be held accountable of these things, where as an AI system cannot be held accountable as it is not a human.

Accepting unchecked AI systems at scale as the future is plain fantasy in the view of regulator, especially in very high risk industries which is why it makes no sense for anyone to trust these systems without any assistance.

[flangola7]

At least for legal there is far more potential than just summarization. Harvey is already producing legal documents with error rates lower than humans.

[rvz]

> Harvey is already producing legal documents with error rates lower than humans.

It is mostly useful and safer for human legal professionals and experts since they have the expertise to check the output but risky and unsafe for those who have little to no legal knowledge.

A user who is a non-legal expert could get into serious trouble if the AI hallucinates output that is contradictory or harms them legally more than it helps them or even both. That is the evergreen risk.

Either way, someone will have to check over the AI's output for that risk and that is for legal human professionals to do, hence why those with no legal experience still trust human lawyers to pay them to check these legal documents.