[obscurette]

Being close relative for several med and care workers we have discussed it a lot and consensus is that it really depends. For example relying on LLM summaries sounds great until it doesn't. It doesn't matter whether you misunderstand LLM summary or LLM "misunderstands" you – there are real risks involved, and you wouldn't want them to weigh on your conscience if they were to materialize.

Relying on LLM to summarize things for you has one more issue. To outsiders, this seems like a tedious process, but is actually very important part of the thought process. Wording your thoughts and writing these down helps people to discover new aspects of the problem. It's how people learn.

At the moment consensus is that it must not be banned, but also not mandated in any way - people must take responsibility, and they must be able to decide for themselves where and when the LLM use is justified and where it is not.

[adrianN]

I don’t think that it is possible to both allow the use of LLM and not mandate them in modern metric driven work places. Either you ban them or you force people to use them for game theoretic reasons: they are slower than their peers and quality of the work is harder to measure than quantity. All you achieve is shifting the blame to the employees if the LLM messes up. Come to think of it, that probably is a highly desirable outcome for the decision makers, so perhaps that will actually be the policy that becomes universally adopted.

[KronisLV]

> I don’t think that it is possible to both allow the use of LLM and not mandate them in modern metric driven work places.

What I’d personally be most concerned about would be the risk of bad models instead of SOTA being used which would be especially error and hallucination prone.

If you’re gonna do it, do it right. Otherwise don’t bother at all.

[squigz]

> All you achieve is shifting the blame to the employees if the LLM messes up.

How is this not a good thing for everyone?

[grayhatter]

> It doesn't matter whether you misunderstand LLM summary or LLM "misunderstands" you – there are real risks involved, and you wouldn't want them to weigh on your conscience if they were to materialize.

"fun" fact, a number (some? many? all?) of the LLM speech to text models will editorialize or hallucinate words for you, to make your speech fit the pattern it expects.

I don't think it's appropriate to use a model that can editorialize in a medical transcription setting. I feel that crosses a huge number of ethical lines.

> At the moment consensus is that it must not be banned, but also not mandated in any way - people must take responsibility, and they must be able to decide for themselves where and when the LLM use is justified and where it is not.

I have a really hard time understanding how supposed experts in their field, have been taken in so perfectly and so completely. You're a trained and certified medical/healthcare worker, responsible for the health, safety and well being of other humans, and yet willingly turn over parts of the process that exists to protect the safety and health of others, to a machine that you know can and will make mistakes you wouldn't or even couldn't ever reasonably make. You're supposed to make sure it works and is safe for everyone you're responsible for before you start using it... but then, I prefer healthcare that doesn't move fast, and break humans.

I'm sure someone will try to play devil's advocate and suggest it's a good thing to ration care, and that you have to acknowledge that you can't save everybody, faster is better after all But no, I'm angry enough about it that if a licensed person suggests that, I'll gladly complain to your licensing board (rhetorically speaking, I'm not gonna try to doxx anyone). You can't ethically argue for rationing care, and letting a machine known to make errors in ethics or care decisions, that you have been trained not to make. (Advocating for a reduction in the standard of care in order to increase profits.) (Yes, you're correct, if you're in a state of triage, you're required to, and in those cases, using an imperfect machine is possibly preferable, and probably ethical, but still objectionable. I currently refuse to believe that the US is in a state of emergency, but perhaps that's where I'm wrong) Part of your licensure you promised to prioritize patient care above personal gain. You can't then make decisions to experiment with systems you cant prove are safe.

I can understand in when SWEs ignore the best interests of the humans they're working for. But Doctors and Nurses? I guess I expected better from my former colleagues?

[nradov]

In a medical setting we first have to be clear about whether the LLM is acting as a transcriptionist or a scribe. Those are different roles. A transcriptionist merely writes down everything the clinician says verbatim, with perhaps a bit of formatting to fit a standard template. Speech recognition software has been used for transcription since before LLMs even existed. Sometimes a human transcriptionist will review and edit the software output. But ultimately the clinician is accountable for approving the document before it legally becomes part of the patient chart.

A medical scribe isn't just taking dictation, they're doing some level of interpretation and often entering structured, coded data directly into an EHR. This is a more complex role and requires some clinical skills. There are some new LLM products that automate this to an extent, but ultimately the licensed clinician is still legally accountable for what goes into the patient chart.

I can absolutely ethically argue for rationing care. All healthcare systems ration care although the means vary. Demand is effectively infinite, especially from older patients with complex or terminal conditions. Resources are finite.

[giardini]

I saw someone whose doctors were misled by this "transcriptionist vs scribe" difference in an ER situation. The LLM was listening and recording as the doctor asked questions and the patient answered. PT had suffered a falling episode. ER doctor asked if pt had any warning (e.g., was dizzy, faint, or unbalanced before fall). Pt said no. Later when reading final discharge report with pt we noticed that the LLM had inserted that the pt "experienced dizziness and then fell" which completely contradicted the pt's actual words.

Since then pt has had to re-explain to multiple doctors how the ER report was incorrect, to their usual MD skepticism. So, for now, I figure LLMs are for shite. Maybe in another 15 years...

And it pays to read everything on a medical report before leaving the facility.

[phoghed]

> I have a really hard time understanding how supposed experts in their field, have been taken in so perfectly and so completely.

Yes, surely it’s not you. It’s all the experts that are wrong and being hoodwinked. Why can’t they just see what you can?

[grayhatter]

> Why can’t they just see what you can?

Most humans use a default trust model for life; they've been lied to. That's part of why I'm angry, if you trust other people; why would you look for it? This entire article is about the fallout from not looking for it, and the resulting harms to pt care. So, it turns out, it was a mistake to adopt the new tech quickly; you know, as evidenced by the harms to patients?

So I don't know, why do you think they couldn't they predict the harms that I thought were obviously predictable, and then happened?

[red75prime]

> where and when the LLM use is justified and where it is not

while being bombarded with articles like "AI makes things worse", "AI consumes all the water" and the like