[mexicocitinluez]

This resonates with what I'm seeing in healthcare right now and the bad taste it's leaving.

There are a whole host of new EMRs popping up that aim to help clinicians make judgement calls about how to answer certain questions and even when particular procedures are relevant. The last one we demoed, each piece of information it retrieved from the LLM has a confidence score attached to it. Our nurses have to fill out 200+ question forms when taking on a new patient that all HAVE to be completed in a single-go meaning that you can't split it up into multiple forms and it's gotta be one cohesive unit.

Imagine being a nurse with little technical skill and almost no idea how these tools work trying to make sense of what the difference between a 90% and 70% is across 200 different questions. "We're 60% sure the patient is allergic to nuts" means jack shit to them. Granted, sometimes the scores are complimented with actual references in the underlying documentation (history and physical, referring info) but sometimes it's not.

[t3hTao]

Healthcare still uses fax. No surprise they arent using Frontier AI to answer these questions. (And if you still don't trust AI, have the report on the left side of the screen and the source pages on the right side of the screen)

[acdha]

If you work in a field which matters, not trusting even “Frontier AI” is also known as “basic professional competence”. If you’re vibe-coding a marketing site nobody dies if you miss a mistake. The same is not true in healthcare or real engineering, and because AI systems can’t be held accountable you will be.

[inspectorSlap]

these can work as long as the evidence is attached (to your point). And in healthcare absolutely must (your EMR example is terrifying). The patient may be allergic to nuts (confidence .84) >> cite: 1) lab value X, PRO Y, ... >> otherwise I'm 62% confident I'm an elephant.