[vpianykh]

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state.

I do not understand why one could commit suicide over something a computer told them. At the same time, I understand that people may be in an unstable state or undereducated to be in a "relationship" with an AI model. I think it's time for humanity to start developing mental health. Otherwise, we are doomed to be a strange hybrid with computer models.

[theGeatZhopa]

I think it is not like "do it now xoxo", but rather ampflification of already strong tendency or depression leading to manifistation of such thoughts and possible actions during long and deep discussions - which LLMs are very good at. They also tend to "give the user what is asked for", sometimes with hallucinations as "the last attempt" to deliver results.

Imagine, you have an addiction towards bets and casino. This kind of addiction is linked with rising dopamine levels when the bet works out. And then you always want to have that winner feeling. Now you talk to the chatbots, since days, weeks, months - they are designed to be a friend/girfriend-alike. And at some point you talk about what you like, some private stuff, ..etc.. you don't need to fall in love with the AI model, but at some point you'll tell it while placing number 2 "you feel lonely lately" and BAM the model may answer you "may be you can go to next casino that will make you feel better" or something like this. This triggers an reaction in your mesolimbic system and you're hooked.

With suicide, there also may be some triggers - I'm not from this field - but, observable is, reports about suicide / news coverage / always put good visible informational text "If you having thoughts .. blabla..call +1-NO-TO-DEATH".

So, if such reports can be triggers, then a model, advised to be your friend or more, can at times be very much a bigger trigger. It may know your deepest thoughts, if you shared them before - which isn't impossible, but rather part of the model design.

May be some chatbot will come up with the idea how to combat addiction, depression and psychological conditions with the power of prompting. But now, its still a world wide problem. I also can't imagine why people start to take hard drugs or do harmfull things to self and each other, but they do - so it may also happen that someone commit suicide. Its better to regulate those fker mdels.