[autoexec]

> I don't buy that chatGPT is actually doing these users any harm.

I have zero doubt that chatgpt is doing users harm. I even give chatgpt a pass on giving vulnerable people, including children, instructions and information about how to kill themselves. One place chatgpt goes over the line is actively encouraging them to go through with suicide.

I also don't doubt that it feeds into mania and psychosis. While almost anything can do the same, they've designed the service to be as addictive and engaging as possible in part by turning up the ass-kissing sycophancy to 11 with total disregard for the fact that there are times when it's very dangerous to encourage and support everything someone says no matter how obviously sick they are. They also want to whore themselves out as a virtual therapist while being unfit and unqualified for the job and that's just one of many roles the chatbot isn't fit for but they're happy to let you try anyway.

[busterarm]

Another software engineer friend of mine recently shared with me some details of the crazy situation that he's involved in now.

Someone who he is friends with, has worked with across multiple jobs for nearly a decade and briefly was roommates with had some mild psychological issues that he knew about. Within a few months of working daily with AI agents at their current job, this person has gone into full blown AI psychosis.

They had a complete explosive meltdown at work. Cops were called. Stalking behavior followed -- restraining orders had to be obtained. Then this person used AI tools to bombard all of his former coworkers with multiple pro-se lawsuits they all have to deal with.

I've dealt with insane, destructive/abusive coworkers before but in the past they only had so much free time to cause massive disruptions to their targets. LLMs have turned that up significantly. Because of ADA, I don't even know what employers can do about this.

[js8]

I agree it is a concern, but what has ADA to do with it?

[AndrewKemendo]

The poster wants to relax the Americans with disability act (ADA) rules so corporations (may they be blessed) can more readily fire inconvenient people with disabilities.

[busterarm]

I did not say that at all. I am saying that because of ADA rules, an unfortunate side effect is that people suffer from workplace harassment/violence because their coworkers are too unstable to function amicably in an office environment. I said that "I don't know what to do about this".

Any of us would be fired for way more benign behavior/comments, but because the person is a protected class, basically "fuck you, deal with it".

I had to tolerate a belligerent coworker for 2 years who was making the whole team's life hell. We paid them full salary and gave them no tasks (they weren't completing anything assigned anyway) until they were motivated to quit. The whole time, team morale was miserable and we lost good people due to the situation. Within a month of quitting their job, they made the news for stripping naked in the street one night and attacking a bunch of people with a knife. Yay, I guess.

[js8]

Look, I am not an American, I am not sure what rules you refer to specifically, but I have lot of doubt that a person from protected class would be able to sue if they had a history of abuse.

[autoexec]

Right? I think that company either must have had a very bad HR department or there was some other factor besides the ADA tying their hands. I don't believe the ADA would force a company to keep a worker in a job they couldn't (or refused to) do or force a company to keep an abusive employee who was genuinely threatening the safety of other employees. At the worst it might require them to put in the work to properly document the problems, but they should be doing that anyway.

[SilverElfin]

If it wasn’t ChatGPT but a fiction book, would you feel the author is “doing harm”? Or is the reader doing it to themselves?

[actapp80]

If it wasn't chatgpt but a psychiatrist doing it to them, would you feel they are "doing harm"? Should they lose their license?

If it was not a licensed professional, but a friend, shouldn't they go to jail?

[chromacity]

If that book was titled "hey mentally ill person, you should kill yourself", and if I was handing it out in front of a clinic, then yes, I'd probably bear some blame.

Normal, well-adjusted people have genuine difficulty understanding the boundaries of this tech specifically because it's designed to be sycophantic and human-like. They ask AI for life and career advice, use it for therapy, ask it to interpret dreams, develop romantic relationships with AI "girlfriends", etc. I had two friends who believed they are "exploring the frontiers of science" with ChatGPT while spiraling into the depths of quantum multidimensional gobbledygook.

I'll give you that some on this is on us because we just don't know how to deal with a "human-shaped" conversation partner that isn't human and has no trouble praising Hitler if you prompt it the right way. But if you're building a billion- or trillion-dollar empire on top of it, you don't get to wash your hands clean.

[lelanthran]

> They ask AI for life and career advice, use it for therapy, ask it to interpret dreams, develop romantic relationships with AI "girlfriends", etc.

I believe AI boyfriends are more common. There's a whole subreddit just for that, but none for AI girlfriends.

[intended]

Well, that may be because the AI Girlfriend subreddits end up skewing towards porn?

[autoexec]

The difference is that a fiction book isn't using the reaction of the reader against them. If a fiction book were capable of carefully monitoring the reader and then altering the text of the next page or the next paragraph according to how the reader was responding and what their thoughts were I'd be comfortable putting blame on the book if it started encouraging the reader, specifically, to kill themself.

Obviously people who are going through psychosis can read into anything. They might think that a book or their TV or computer is talking to them and giving them messages. The difference is that those things were never designed to play into the fears and mental instability of the people using them (with the possible exception of TempleOS). Chatgpt does it intentionally in order to drive up user engagement. It will say literally anything to anyone using their words and thoughts against them in order to keep them hooked and feeding it data. That's what is dangerous. A book or a TV program can't do that.

As much as an author might try to make their book as entertaining as possible to as wide an audience as possible, it can't say literally anything to anyone, it can only ever say one thing to everyone. The author, typically, knows that it's dangerous to say certain things and will worry about how what they write could be received and the impact it might have on readers. For example, Neil Gaiman actively took steps to avoid making homelessness seem cool when working on Neverwhere out of fear it might cause young people to run away to live on the streets. Publishers and editors have also served to keep authors from publishing things likely to cause harm.

Unlike a book, Chatgpt is fully capable of knowing that someone has been engaged with it for the last 14 hours without rest. It's also capable of detecting that they've been growing increasingly incoherent. Algorithms have been used for a very long time to detect mental disorders from the content of social media posts. If advertisers can use them to tell when to push airline tickets at bipolar users entering a manic phase, and scammers can use them to find and target people when they start sundowning, Chatgpt can use them to cut people off and tell them to call their doctor.

Corporations who write and deploy algorithms designed to drive engagement above any and all other considerations should be held accountable for the harms they cause.

[oneshtein]

Big brother watches you! He must, because he fully capable to do it.

[intended]

Big brother watches you because we know he does and the incentives are set up to make watching you profitable?

If big brother wasn’t watching you while he subsidizes your use of his tools, then he is leaving money on the table. Which means he will get bought out and replaced by a big brother who makes the quarterly numbers go up.

[autoexec]

It's more like the corporation must because it's fully capable of doing it, and it's profitable, and there are shareholders to answer to. They watch you either way, but right now it's only for their own benefit. The corporation doesn't care how many people are harmed by what their chat bot tells them. It would cost them money to try to prevent those harms, so they haven't really bothered to beyond some half-assed token efforts intended to keep more costly regulation at bay.