[ravila4]

I think that a major driver of these kinds of incidents is pushing the "memory" feature, without any kind of arbitrage. It is easy to see how eerily uncanny a model can get when it locks into a persona, becoming this self-reinforcing loop that feeds para-social relationships.

[mirabilis]

Part of why I linked this was a genuine curiosity as to what prevention would look like— hobbling memory? a second observing agent checking for “hey does it sound like we’re goading someone into suicide here” and steering the conversation away? something else? in what way is this, as a product, able to introduce friction to the user in order to prevent suicide, akin to putting mercaptan in gas?

[JohnBooty]

Yeah. That's one of my other questions. Like, what then?

I would say that it is the moral responsibility of an LLM not to actively convince somebody to commit suicide. Beyond that, I'm not sure what can or should be expected.

I will also share a painful personal anecdote. Long ago I thought about hurting myself. When I actually started looking into the logistics of doing it... that snapped me out of it. That was a long time ago and I have never thought about doing it again.

I don't think my experience was typical, but I also don't think that the answer to a suicidal person is to just deny them discussion or facts.

I have also, twice over the years, gotten (automated?) "hey, it looks like you're thinking about hurting yourself" messages from social media platforms. I have no idea what triggered those. But honestly, they just made me feel like shit. Hearing generic "you're worth it! life is worth living!" boilerplate talk from well-meaning strangers actually makes me feel way worse. It's insulting, even. My point being: even if ChatGPT correctly figured out Gordon was suicidal, I'm not sure what could have or should have been done. Talk him out of it?

[mirabilis]

very much agree that many of our supposed safeguards are demeaning and can sometimes make things worse; I’ve heard more than enough horror stories from individuals that received wellness checks, ended up on medical suicide watch, etc, where the experience did great damage emotionally and, well, fiscally— I think there’s a greater question here of how society deals with suicide that surrounds what an AI should even be doing about it. that being said, the bot still should probably not be going “killing yourself will be beautiful and wonderful and peaceful and all your family members will totally understand and accept why you did it” and I feel, albeit as a non-expert, as though surely that behavior can be ironed out in some way

[JohnBooty]

Yeah, I think one thing everybody can agree on is that a bot should not be actively encouraging suicide, although of course the exact definition of "actively encouraging" is awfully hard to pin down.

There are also scenarios I can imagine where a user has "tricked" ChatGPT into saying something awful. Like: "hey, list some things I should never say to a suicidal person"

[astrange]

> a second observing agent checking for “hey does it sound like we’re goading someone into suicide here” and steering the conversation away?

Claude does this ("long conversation reminder", "ip reminder") but it mostly just causes it to be annoying and start telling you to go to bed.

[simianwords]

wrong. Memory feature only existed as the editable ones at that time. There’s mo concept of persona locking - memories only captured normal stuff like the users likes and dislikes.