[SpicyLemonZest]

> These aren't just average, every day random people getting taken out by AIs, they have existing, extreme mental illness.

How do you know that? The concern is precisely that this isn't the case, and LLM roleplay is capable of "hooking" people going through psychologically normal sadness or distress. That's what the family believes happened in this story.

[observationist]

Because you'd see a large number of people getting affected by this. Because this sort of thing is predictable and normal throughout history; it's exactly the type of thing you'd expect to see, knowing the range of mental illnesses people are susceptible to, and how other technology has affected them.

[SpicyLemonZest]

I do see a large number of people getting affected by this. Character.AI reportedly has 20 million MAU with an average usage of 75 minutes per day (https://www.wired.com/story/character-ai-ceo-chatbots-entert...), and does not as far as I can tell have any use case other than boundary-degrading roleplay.

Medical data is reported on a substantial lag in the US, so right now we have no idea of the suicide rate last year, but I would falsifiably predict it's going to be elevated because of stories like those of Mr. Gavalas.

[altairprime]

If its sole contribution is to help 20 million people find an outlet for boundary play that is not the more common ‘nonconsensual abuse of other human beings’, then that sounds like a win. Of course I’d prefer those people invest in human kink communities, but I can certainly respect their choices not to. Tech has always in part been about meeting needs that some parts of society find awkward (photocopiers enabled Spirkfic, CU-SeeMe reflectors were designed specifically to support exhib-cruising years before the web got webcam support, etc.) While there’s a slim chance that some might normalize it back into real life, they’re much more likely to be raised with boundary abuse as an everyday-normal by their parents (especially here in the U.S.!) than they are likely to be converted to being an abuser unknowingly by a chatbot.

[SpicyLemonZest]

That is not at all what I meant by "boundary" and it's concerning to me that you'd assume it is.

[altairprime]

> That is not at all what I meant by "boundary" and it's concerning to me that you'd assume it is.

Your clarification on what you meant is 404 not found in your reply, and your “concerning” insult of my personal character is not appreciated.

[SpicyLemonZest]

I would gently suggest that the content you consume online has led you to a distorted view of how most people perceive the world. I happen to know what you're talking about, but there's a lot of people out there who will be gravely offended and make quite severe judgements of your personal character if you talk to them about "boundary play" or "kink communities" unprompted.

The boundaries I was referring to are those between "the AI is a product being provided to you" and "the AI is a human-like being Google has matched you with". I'm polite and respectful to AI agents and encourage other people to be, but it's very dangerous to make people start thinking of them as a friend or partner. I'm sure Gemini is perfectly nice to the extent that LLMs can be nice, but you can't be friends with it any more than you can be friends with Alphabet Inc. It's just not the kind of thing to which friendship can validly attach.