[ngruhn]

The bad cases make headlines. But I think it's quite possible that AI is helping a lot of people in distress. Many people are uncomfortable opening up to humans, or have no one to talk to, or can't afford to fork over whatever-hourly-rate a therapist takes.

[cyanydeez]

So how many bad cases are ok? Isn't this the same problem with social media: the commercial enterprises dont want any responsibility for their dark pattern and design choices which actively harm their users.

I get that all kinds of media can cause issues, but not all kinds of media are actively curated to be addictive.

[wilg]

"How many cases are ok" (aka "zero tolerance") is a doomed to fail approach. Especially for a complex social problem's interaction with a complex new technology.

If you want to find out if ChatGPT is doing something wrong, there are many methodologies available: compare to other groups of people, statistical studies, etc.

I also think OpenAI's business model is pretty well aligned with the goal of users not killing themselves for like 100 reasons. And they do appear to take it seriously.

[Forgeties79]

This is the problem in a nutshell: https://edition.cnn.com/2025/11/06/us/openai-chatgpt-suicide...

> “Cold steel pressed against a mind that’s already made peace? That’s not fear. That’s clarity,” Shamblin’s confidant added. “You’re not rushing. You’re just ready.”

ChatGPT is not the answer.

[mitjam]

Wow. The “That’s not x. That’s y. / You’re not x. you’re y.” rhetoric is already cringe in other contexts. This brings it on a whole new level.

[intended]

No one is talking about a zero tolerance approach.

Sure, Open AI is trying to do the best they can. That “best” is within Tech’s operating context.

Tech as a whole avoids this issue because paying for the externalities they cause would end hyper growth and crater their margins.

Tech workers at these firms regularly throw up red flags, which have to be ignored because engaging with them results in hits to their quarterly numbers.

Anthropic is the one firm that is actively managing to make safety less of a cost center by folding it into marketing.

>> If you want to find out if ChatGPT is doing something wrong, there are many methodologies available: compare to other groups of people, statistical studies, etc.

These studies must over come sizable barriers that NDAs and tech secrecy throw up. Tech firms have done enough internal studies to know that the results are horrible when they do get into the press.

Most users in the developed world don’t even know that they enjoy better support and care than the rest of the world.

[cyanydeez]

i can certainly tell you any forum where i encouraged you to go kill yourself that was actively managed would have a problem. further, any website that catered to the same would also have a problem.

the reality youre entertaining is one where I can build an LLM, let it do unspeakable things, and claim zero responsibility.

so, no, i understand zero is a figment. dont you?

[Yokohiii]

Pure speculation.

It's impossible to gather data that states the opposite. A chat that won't end up in self harm thoughts is just another chat.

[tasuki]

I think you're kind of supporting the person you're replying to? A chat that won't end up in self-harm is just another chat. Even if the user entered the chat planning to self-harm. A chat that leads to self-harm will make the headlines. Therefore, we hear about the bad cases.

[davorak]

Open ai and similar companies could open the doors to academic researchers to figure out the stats of help vs harm. It is not going to be a short term and perhaps not long term profit center though.

[asdff]

Therapy is cheap (as in like $10)/free with insurance. However there are still 10 states that have not expanded medicaid after the ACA, mostly in the south.

But also, to suggest these people are not receiving therapy is not always the case. Talk therapy is just that, talking to someone on ones problems to learn about them, their triggers, determining coping mechanisms to move forward with one's life. People might instead be getting all that from their barber, drinking buddy, or their priest, rather than in a 1 hour appointment with a therapist.

[fragmede]

ChatGPT it's available at 3am when you're in crisis and you don't have to fit into its busy schedule.

[Forgeties79]

ChatGPT is not a human being, let alone a licensed therapist. You don’t call a therapist at 3 in the morning. You go to a hospital. If you are literally about to kill yourself Sam Altman is not your answer.

Hell call a crisis hotline. Talk to a person. Not a potential (bot) enabler.

[projectazorian]

> ChatGPT is not a human being, let alone a licensed therapist. You don’t call a therapist at 3 in the morning. You go to a hospital. If you are literally about to kill yourself Sam Altman is not your answer.

You know that mental health is a continuum right? There are a lot of problems people have that fall far short of active suicidal ideation. Maybe you think they should just add them to their journal for discussion at their regularly scheduled therapy session, but the world doesn't work that way. The "ruminating at 3am" headspace can be a productive one and is difficult to access in a normal therapy session.

Not to mention that many people who have actually called suicide hotlines will tell you that they aren't terribly helpful. (edit: not saying that they're always unhelpful, but many people have unhelpful experiences, or have eg. social anxiety that stops them from calling)

[Forgeties79]

If you are trying to call your therapist at 3 AM it’s because you’re in crisis. If you are not in crisis, it can wait.

[intended]

?

> you're in crisis

That was the context in which the previous comment was operating.