[Aurornis]

> It seems from the reports many people are being affected

I think the rapid scale and growth of ChatGPT are breaking a lot of mental models about how common these occurrences are.

ChatGPT's weekly active user count is twice as large as the population of the United States. More people use ChatGPT than Reddit. The number of people using ChatGPT on a weekly basis is so massive that it's hard to even begin to understand how common these occurrences are. When they happen, they get amplified and spread far and wide.

The uses of ChatGPT and LLMs are very diverse. Calling for a shutdown of long conversations if they don't fit some pre-defined idea of problem solving is just not going to happen.

[miltonlost]

Ah, the old "we're too big to be able to not do evil things! we've scaled too much so now we can't moderate! Oh well, sucks to not be rich."

[Aurornis]

They're not claiming they don't moderate, though. Where are you getting that? A common complaint about ChatGPT and even their open weights models is that they're too censored.

[fragmede]

Anthropic at least used to stop conversations cold when they reached the end of the context window, so it's entirely possible from a technical standpoint. That OpenAI chooses not to, and prefers to let the user continue on, increasing engagement, puts it on them.

[michaelmrose]

Incidence of harm is a function of harm/population. It is likely that Facebook is orders of magnitude more harmful than ChatGPT and bathtubs and bikes more dangerous than long LLM conversations.

It doesn't mean something more should not be done but we should retain perspective.

Maybe they should try to detect not long conversations but dangerous ones based on spot checking with a LLM to flag problems up for human review and a family notification program.

EG Bob is a nut. We can find this out by having a LLM not pre prompted by Bob's crazy examine some of the chats by top users by tokens consumed in chat not API and flagging it up to a human who cuts off bob or better shunts him to a version designed to shut down his particular brand of crazy eg pre prompted to tell him it's unhealthy.

This initial flag for review could also come from family or friends and if OpenAI concurs handle as above.

Likewise we could target posters of conspiracy theories for review and containment.

[yousif_123123]

> Calling for a shutdown of long conversations if they don't fit some pre-defined idea of problem solving is just not going to happen.

I am calling for some care to go in your product to try to reduce the occurrence of these bad outcomes. I just don't think it would be hard for them to detect that a conversation has reached a point that its becoming very likely the user is becoming delusional or may engage in dangerous behavior.

How will we handle AGI if we ever create it, if we can't protect our society from these basic LLM problems?

[sendes]

>its becoming very likely the user is becoming delusional or may engage in dangerous behavior.

Talking to AI might be the very thing that keeps those tendencies below the threshold of dangerous. Simply flagging long conversations would not be a way to deal with these problems, but AI learning how to talk to such users may be.

[tinfoilhatter]

In June 2015, Sam Altman told a tech conference, “I think that AI will probably, most likely, sort of lead to the end of the world. But in the meantime, there will be great companies created with serious machine learning.”

Do you really think Sam or any of the other sociopaths running these AI companies care whether their product is causing harm to people? I surely do not.

[1] https://siepr.stanford.edu/news/what-point-do-we-decide-ais-...

[shwaj]

It seems like a cheaper model could be asked to review transcripts, something like: “does this transcript seem at all like a wacky conspiracy theory that is encouraged in the use by the LLM”?

In this case, it would have been easily detected. Depending on the prompt used, there would be more or less false positives/negatives, but low-hanging fruit such as this tragic incident should be avoidable.