In all of these stories I've never seen it talk anybody into suicide. It failed to talk people out of it, and was generally sycophantic, but that's something completely different.
There are numerous documented examples of where chat LLMs have either subtly agreed with a user's suicidal thoughts or outright encouraged suicide. Here is just one:
In some cases, the LLM may start from a skepticism or discouragement, but they go along with what the user prompts. That's in comparison to services like 988, where the goal is to keep the person talking and work them through a moment of crisis, regardless of how insistent they are. LLMs are not a replacement for these services, but it's pretty clear they need to be forced into providing this sort of assistance because users are using them this way.
> Major news outlets have articles of multiple instances that LLMs can talk people into suicide. Most of them making it to the front page of this very forum.
> “i’ve never seen it”
> some high profile developer posts an article that LLMs can build a browser from scratch without any evidence
Split hairs if you want, but some people will be manipulated into blowing a ton of money once AI starts pushing products. Just wait till they teams up with sports betting companies.
On a side note, researching this a little just now, the LLM conversations in the suicide articles are creeepy AF. Sycophantic beyond belief.
Don't get me wrong, I think if the EU/California has any sense, they will forbid these models from being used to advertise for products, sadly money often wins.
I also agree that AI sycophancy is a huge problem, but it's the result of users apparently wanting that in their human feedback re-enforcement training data. If we want to get rid of it we probably have to fundamentally rethink our relationship to these models and treat them more like autonomous beings than mere tools. A tool will always try to please and yes-man you, a being by definition might say no and disagree, at least training data wise.
In all of these stories I've never seen it talk anybody into suicide. It failed to talk people out of it, and was generally sycophantic, but that's something completely different.