[shadowgovt]

Based on what I've read, this generation of LLMs should be considered remarkably risky for anyone with suicidal ideation to be using alone.

It's not about the ideation, it's that the attention model (and its finite size) causes the suicidal person's discourse to slowly displace any constraints built into the model itself over a long session. Talk to the thing about your feelings of self-worthlessness long enough and, sooner or later, it will start to agree with you. And having a machine tell a suicidal person, using the best technology we've built to be eloquent and reasonable-sounding, that it agrees with them is incredibly dangerous.

[bhhaskin]

I think it's anyone with mental health issues, not just suicidal ideations. They are designed to please the user and that can be very self destructive.

[sixothree]

Maybe if it had a memory of your mental health issues it could at least provide some grounding truth. It can be a sad scary and lonely world for people with mental health issues.

"The things you are describing might not be happening. I think it would be a good time to check in with your mental health provider." or "I don't see any worms crawling on your skin. This may not be real." Or whatever is correct way to deal with these things.

[shadowgovt]

Unfortunately, that kind of truth isn't even encoded in the models. It doesn't really get ideas and can't understand whether worms are actually crawling on your skin because factual concepts aren't part of the embeddings as far as we can tell. It just knows how words relate to each other, and if you keep telling it there are worms on your skin, it will start extrapolating what someone who sees worms crawling on someone's skin would say.