[qwertylicious]

This is the story of the modern tech industry at large: a major new technology is released, harms are caused, but because of industry norms and a favourable legal environment, companies aren't held liable for those harms.

It's pretty amazing, really. Build a washing machine that burns houses down and the consequences are myriad and severe. But build a machine that allows countless people's private information to be leaked to bad actors and it's a year of credit monitoring and a mea culpa. Build a different machine that literally tells people to poison themselves and, not only are there no consequences, you find folks celebrating that the rules aren't getting in the way.

Go figure.

[moduspol]

I think the harms of expensive and/or limited and/or inconvenient access to even basic medical expert Q&A are far greater. Though they're not as easy to measure.

[tempodox]

And, what, LLMs to the rescue?

[rschneid]

I think their point is that, in general, social-scale healthcare is an under-solved problem in practice and LLMs have potential to improve a significant portion these challenges by increasing accessibility to treatment. The availability of these tools will inevitably lead to more instances of reports like this (from the report the article is based on):

> This case also highlights how the use of artificial intelligence (AI) can potentially contribute to the development of preventable adverse health outcomes. Based on the timeline of this case, it appears that the patient either consulted ChatGPT 3.5 or 4.0 when considering how he might remove chloride from this diet. Unfortunately, we do not have access to his ChatGPT conversation log and we will never be able to know with certainty what exactly the output he received was, since individual responses are unique and build from previous inputs.

However I don't see this single negative instance of a vast social-scale issue as much more than fear/emotion-mongering without at least MENTIONING that LLM also have positive effects. Certainly, it doesn't seem like science to me. Unless these models are subtly leading otherwise healthy and well-adjusted users to unhealthy behavior, I don't see how this interaction with artificial intelligence is any different than the billions of confirmation-bias pitfalls that already occur daily using google and natural stupidity. From the article:

> The case also raises broader concerns about the growing role of generative AI in personal health decisions. Chatbots like ChatGPT are trained to provide fluent, human-like responses. But they do not understand context, cannot assess user intent, and are not equipped to evaluate medical risk. In this case, the bot may have listed bromide as a chemical analogue to chloride without realizing that a user might interpret that information as a dietary recommendation.

It just seems they've got an axe to grind and no technical understanding of the tool they're criticizing.

To be fair, I feel there's much to study and discuss about pernicious effects of LLMs on mental health. I just don't think this article frames these topics constructively.

[usrnm]

How many people do you think the early steam engines killed? Or airplanes

[qwertylicious]

Or sweatshops or radium infused tinctures.

We've moved on from the 1800s. Why are you using that as your baseline of expectation?

[api]

There's a very common belief that things like regulations and especially liability simply halts all innovation. You can see some evidence for this point of view from aerospace with its famous "if it hasn't already flown, it can't fly" mentality. It's why we are still using leaded gasoline in small planes, though this is finally being phased out... but it took an unreasonably long time due to certification requirements and bureaucracy.

If airplanes weren't so heavily regulated we'd have seen leaded gasoline vanish there around the same time it did in cars, but you also might have had a few crashes due to engine failures as the bugs were worked out with changes and retrofits.

I'm a little on the fence here. I don't want a world where we basically conduct human sacrifice for progress, but I also don't want a world that is frozen in time. We really need to learn how to have responsible, careful progress, but still actually do things. Right now I think we are bad at this.

Edit: I think it's related to some extent to the fact that nuanced positions are hard in politics. In popular political discourse positions become more and more simple, flat, and extreme. There's a kind of context collapse that happens when you try to scale human organizations, what I want to call "nuance collapse," that makes it very hard to do anything but all A or all B. For innovation it's "full speed ahead" vs "stop everything."

[pjc50]

Yes. It's also worth thinking about the sharp cliff effect. Things either fall into the category of "medical device" (expensive, heavily regulated, scarce, uninnovative), or they don't, in which case it's a free for all of unregulated supplements and unsupported claims.

The home brew "automatic pancreas" by making a bluetooth control loop between a glucose monitor and an insulin pump counts as a "medical device". Somehow a computer system that encourages people to take bromide isn't. There ought to be a middle ground.

[bigbadfeline]

> Somehow a computer system that encourages people to take bromide isn't. There ought to be a middle ground.

Yes, there is a very effective middle ground that doesn't punish anybody for providing information. It's called a disclaimer:

"The information provided should no be construed as medical advise. Please seek other sources of information and/or consult a physician before taking any supplements recommended by LLMs or web sites. This bot is not responsible for any adverse effects you may think are due to my information"

When an LLM model detects a health related question - print the above disclaimer before the answer.

There is no need for dictatorship in order to save people from information.

[qwertylicious]

It's also called "liability".

"Warning, this washing machine might burn your house down" is not sufficient to escape punishment. Why should digital technology get a pass just because the product that's offered is intangible?

[api]

Learning to innovate steadily and responsibly without just stopping is one of the things I'd put on my list of things humanity needs to figure out.

Individuals can do it, but as I said it doesn't scale. An individual can carefully scale a rock face. A committee, political system, or corporate board in charge of scaling rock faces would either scale as fast as possible and let people fall to their deaths or simply stand at the bottom and have meetings to plan the next meeting to discuss the proper climbing strategy (after discussing the color of the bike shed) forever. Public discourse would polarize into climb-fast-die-young versus an ideology condemning all climbing as hubris and invoking the precautionary principle, and many door stop sized books would be written on these ideas, and again either lots of people would die or nothing would happen.

[bigbadfeline]

From the OP:

> "There may have been multiple factors contributing to the man’s psychosis, and his exact interaction with ChatGPT remains unverified. The medical team does not have access to the chatbot conversation logs and cannot confirm the exact wording or sequence of messages that led to the decision to consume bromide."

Any legal liability for providing information is wrought with opportunities for abuse, so bigly so that it should never be considered.

[qwertylicious]

I have nothing to add other than to say this is, IMO, exactly right. I have no notes.

[throwaway173738]

I think they were suggesting that LLMs are a nascent technology and we’d expect them to kill a bunch of people in preventable accidents before being heavily regulated.

[tim333]

Medical error kills ~300k per year in the US these days. AI might actually help reduce that.

[qwertylicious]

Sure, when applied thoughtfully and judiciously.

Look back. At no point did I suggest AI should be banned or outlawed. My remedy for washing machines burning down houses isn't to ban washing machines. It's to ensure there are appropriate incentives in place (legal, financial, reputational) to encourage private industry to consider the potential negative externalities of what they're doing.

[pjc50]

Quite a lot. Boiler explosions were common until a better understanding was reached of the technology. Is this supposed to be an argument in its favor?

[ineedasername]

Yes, people have died in preventable ways before, so as technology progresses and civilization has advanced in countless ways in the last 200+ years, we should not attempt to improve nor even critique preventable deaths that we either did not or could not before. It should be an area of advancement that we enshrine in status quo as we in other areas rush forward and even race for improvements.

[specproc]

Considerably fewer when regulated.

[Smeevy]

How many people were killed after following medical advice from steam engines and airplanes?

[m463]

it's even easier to point to cars