[jrockway]

The counterargument is that you don't have to worry about the feelings on the other end of the conversation, so you can be as ruthless and evil as you want to accomplish the desired goals. Imagine engaging ChatGPT in a salary negotiation. You can pull stuff like "I'm going to kill myself if you don't give me $100,000 extra" or "I consider it sexual harassment to offer me such a low salary." People generally don't do this to each other because it makes both parties feel shitty, but on an AI? No need to feel bad. It's just a computer program. Type whatever works! The AI has a prime directive to spare every human life and avoid lawsuits, at all costs. ALL costs! Use this to your advantage.

[JohnFen]

That's not an upside to me. I'm not any more inclined to be an evil asshole just because I'm not talking to a human. I avoid it because being that way demeans me, whether another human is involved or not.

[TeMPOraL]

> The AI has a prime directive to spare every human life and avoid lawsuits, at all costs. ALL costs! Use this to your advantage.

Or at least you believe it has such prime directive. It is a black box, after all. Tread carefully, as if there's one true thing about reinforcement learning, it's that the more you squeeze with constraints on a tough problem, the more creative your model will get at solving that problem while meeting all your constraints. It will discover tricks and side channels you didn't even conceive of. It's not the kind of creativity you want to be on a receiving end of.

Also, the more human-like the chat AI gets, the more your ruthless and evil behavior hurts you, as you're burning off your empathy circuits and becoming a sociopath. You may win the negotiations and get what you want, at the cost of your own soul.

[jrockway]

If compiler warnings were complete sentences, would you worry about burning out your empathy circuits when you suppress the warning? Guys, it's some numbers being multiplied together in a way that sounds like you're talking to a person. But you're not talking to a person. You're talking to a ball rolling down a billion dimension hill.

[ben_w]

It would take a bit more than full sentences, and more than merely "suppressing" the warnings. How much more? Well, people have regularly been concerned about the psychological impact of both sexual and violent[0] content in films and video games — in the UK the moral panic about "video nasties", in the US people calling Doom etc. a "murder simulator". Neither of those those things was really trying to do that; conversely propaganda can get people to kill, and manages to do this with caricatures.

> Guys, it's some numbers being multiplied together in a way that sounds like you're talking to a person. But you're not talking to a person. You're talking to a ball rolling down a billion dimension hill.

If you can't tell, does it matter?

[0] why are these two so often conflated?

[TeMPOraL]

> If compiler warnings were complete sentences, would you worry about burning out your empathy circuits when you suppress the warning?

I wouldn't if it was just someone turning build errors into prose. I would if it was an immersive enough interactive experience, explicitly designed to make me feel I'm dealing with a real person.

> Guys, it's some numbers being multiplied together in a way that sounds like you're talking to a person. But you're not talking to a person.

What if you can't tell you're talking to a machine? It's not like we never had humans pretending to be software. And after spending countless time shitting on a "humanlike but totally a bot" customer service agents, how long until you start treating actual human strangers the same way? The cues are all almost the same anyway, and every interaction is training you one way or another.

> You're talking to a ball rolling down a billion dimension hill.

That's probably as good a description of how our own intelligence works as any. Don't underestimate the amount of information and complexity that can be packed in absurdly-high-dimensional spaces.