[lacoolj]

Not a lawyer.

While AI is not a real human, brain, consciousness, soul ... it has evolved enough to "feel" like it is if you talk to it in certain ways.

I'm not sure how the law is supposed to handle something like this really. If a person is deliberately telling someone things in order to get them to hurt themselves, they're guilty of a crime (I would expect maybe third-degree murder/involuntary manslaughter possibly, depending on the evidence and intent, again, not a lawyer these are just guesses).

But when a system is given specific inputs and isn't trained not to give specific outputs, it's kind of hard to capture every case like this, no matter how many safe-guards and RI training is done, and even harder to punish someone specific for it.

Is it neglect? Or is there malicious intent involved? Google may be on trial for this (unless thrown out or settled), but every provider could potentially be targeted here if there is precedent set.

But if that happens, how are providers supposed to respond? The open models are "out there", a snapshot in time - there's no taking them back (they could be taken offline, but that's like condemning a TV show or a book - still going to be circulated somehow). Non-open models can try to help curb this sort of problem actively in new releases, but nothing is going to be perfect.

I hope something constructive comes from this rather than a simple finger pointing.

Maybe we can get away from natural language processing and go back to more structured inputs. Limit what can be said and how. I dunno, just writing what comes to mind at this point.

Have a good day everyone!

[bluGill]

My companies makes potentially dangerous things like lawn mowers. We have a long set of training on how to handle safety issues that gets very complex. Our rules about safety issues is "design it out, then guard it out, and finally warn it out" - that is an ordered list so we cannot go to the next step until we take the previous as far as we can. (and every once in a while we [or a competitor] realize something new and have to revisit everything we sell for that new idea)

Courts will see these things for a while, but there have been enough examples of this type of thing that all AI vendors needs to either have some protection in their system. They can still say "we didn't think of this variation, and here is why it is different from what we have done before", but they can't tell the courts we had no idea people would do stupid things with AI - it is now well known.

I expect this type of thing to play out over many years in court. However I expect that any AI system that doesn't have protection against the common abuses like this that people do will get the owners fined - with fines increasing until they are either taken offline (because the owners can't afford to run them), or the problem fixed so it doesn't happen in the majority of cases.

[LeifCarrotson]

Is the headline actually surprising to anyone? AI products that are currently live on a half dozen cloud providers are fueling thousands of people's various delusions right now.

No, the LLM itself is not a human, but the people running the LLM are real people and are culpable for the totally foreseeable outcomes of the tool they're selling.

The vendors will argue that the benefits that some people are gaining from access to those tools outweigh the harms that some other people like Jonathan (and like Joel, his father) are suffering. A benefit of saving a few seconds on an email and a harm of losing a life due to suicide are not equivalent. And sure, the open models are out there, but most users aren't running them locally: they're going through the cloud providers.

Same human responsibility chain applies to self-driving cars, BTW. If a Waymo obstructs an ambulance [1] then Tekedra Mawakana, Dmitri Dolgov, and the rest of the team should be considered to have collectively obstructed that ambulance.

[1]: https://www.axios.com/local/austin/2026/03/02/waymo-vehicle-...