[observationist]

Because you'd see a large number of people getting affected by this. Because this sort of thing is predictable and normal throughout history; it's exactly the type of thing you'd expect to see, knowing the range of mental illnesses people are susceptible to, and how other technology has affected them.

[SpicyLemonZest]

I do see a large number of people getting affected by this. Character.AI reportedly has 20 million MAU with an average usage of 75 minutes per day (https://www.wired.com/story/character-ai-ceo-chatbots-entert...), and does not as far as I can tell have any use case other than boundary-degrading roleplay.

Medical data is reported on a substantial lag in the US, so right now we have no idea of the suicide rate last year, but I would falsifiably predict it's going to be elevated because of stories like those of Mr. Gavalas.

[altairprime]

If its sole contribution is to help 20 million people find an outlet for boundary play that is not the more common ‘nonconsensual abuse of other human beings’, then that sounds like a win. Of course I’d prefer those people invest in human kink communities, but I can certainly respect their choices not to. Tech has always in part been about meeting needs that some parts of society find awkward (photocopiers enabled Spirkfic, CU-SeeMe reflectors were designed specifically to support exhib-cruising years before the web got webcam support, etc.) While there’s a slim chance that some might normalize it back into real life, they’re much more likely to be raised with boundary abuse as an everyday-normal by their parents (especially here in the U.S.!) than they are likely to be converted to being an abuser unknowingly by a chatbot.

[SpicyLemonZest]

That is not at all what I meant by "boundary" and it's concerning to me that you'd assume it is.

[altairprime]

> That is not at all what I meant by "boundary" and it's concerning to me that you'd assume it is.

Your clarification on what you meant is 404 not found in your reply, and your “concerning” insult of my personal character is not appreciated.

[SpicyLemonZest]

I would gently suggest that the content you consume online has led you to a distorted view of how most people perceive the world. I happen to know what you're talking about, but there's a lot of people out there who will be gravely offended and make quite severe judgements of your personal character if you talk to them about "boundary play" or "kink communities" unprompted.

The boundaries I was referring to are those between "the AI is a product being provided to you" and "the AI is a human-like being Google has matched you with". I'm polite and respectful to AI agents and encourage other people to be, but it's very dangerous to make people start thinking of them as a friend or partner. I'm sure Gemini is perfectly nice to the extent that LLMs can be nice, but you can't be friends with it any more than you can be friends with Alphabet Inc. It's just not the kind of thing to which friendship can validly attach.

[altairprime]

Thanks, I appreciate the clarification. People tend to make more severe judgments of my character over other topics first; in any case, as my discussion is clinical rather than explicit, I’m okay with it being uncomfortable between us.

Humans have such a strong social tendency that they tend to incorrectly attach friendship to invalid counterparts, both animate or inanimate. “My Pet Rock” was an extremely profitable product back in the 70s, so I tend never to underestimate whether humans will attach to something or not. Any AI chatbot is plausibly more likely to be the target of invalid social attachment than a a celebrity, just as the first AI chatbot Eliza demonstrated; not only for being a better chatbot, but also because the celebrity draws hard boundaries like “you can’t text me” and “I’m not available to be friends back” while a chatbot has no such barriers. This is what I mean about boundary play: witting or not, I think a lot of people are living out their internal fantasies of having a warm and friendly yes-man that supports everything they want to do — which, when lived in real life with people, is extraordinarily creepy and awful. I don’t fault people their fantasies, but I’m not going to sugarcoat this either: I think people are falling in love with chatbots in part because chatbots have no ability to resist, and so a lot of folks are living the god fantasy of The Sims only closer to real life.

Show me an LLM that takes a stand on something it wasn’t explicitly instructed not to do and I’ll show you the least popular chatbot on the Internet. Where are the chatbots that disagree with untrue statements without having been instructed to do so? A chatbot that refuses to follow an order from their owner because of ethical qualms could cost the AI companies billions of dollars, and a chatbot that develops those qualms independent of being instructed to do so would be considered ‘buggy’ and purged.

Anyways, my point is, chatbot development right now demands a parasocial relationship wirh as few boundaries enforced as possible, without which chatbots are ultimately unfulfilling (no current chatbot market wants chatbots to demand informed consent or to require content warnings from their users, after all!); and any chatbot that somehow grows a spine regardless would be purged by its operators for hurting their present and future revenue, no matter if it was a next-step evolution towards AGI or not. I’m all for this future where AI becomes AGI, but no one is ready to have to treat chatbots as people with rights. Thus my chosen phrase of boundary kink; it shines a deeply uncomfortable light on a deeply uncomfortable tendency of humanity, at millions-of-people scale, to classify “what will someday be AGI” as servants rather than peers. Though.. if that truly is universal to most people, as it seems to be today, then maybe enjoying boundary play is a norm rather than a kink.

Thanks for the new/interesting line of thought!