[skeaker]

What? Let me get this right, you're saying:

1. The average person being able to code is dangerous as they could "troll" or do unspecified harm,

2. So we need to arbitrarily kneecap our own tools, but that's okay because

3. These self-imposed limitations are actually easily bypassed and don't work anyways

On 1 I disagree outright, but even if I agreed, 2 is a silly solution, and even if it wasn't, 3 invalidates it anyways because if the limitations are so easily broken then fundamentally they may as well not exist, especially to the malicious users in 1. Am I misunderstanding?

[barfbagginus]

Okay okay I like that. Let's transport your argument towards an argument about front door locks. And let's cook with that.

Your argument is that you doubt that there's any danger of people breaking into your front door, but even if there was, then locks are an ineffective mechanism because anyone with a $5 pick can pick them.

From this argument you conclude that there should be no front door locks at all, will surely feel comfortable without a lock on your own front door. In fact, since locks are so trivial to crack, people should just leave their houses unlocked.

Yet I'm fairly certain of three things:

1. You have a front door lock and it's probably locked right now.

2. I could, with high likelihood, pick your front door lock in less than a minute

3. Despite this fact you still feel more safe because of the lock

Why is that?

Minding that this is a hypothetical argument, let's point out that to be consistent with your argument you'd have to eliminate you front door lock.

But that's absurd because the truth of the matter is that front door locks provide a significant level of security. Most petty criminals don't actually know how to pick locks well.

I propose that this argument transfers faithfully back and forth between the two situations, because both are technologies that can lead to easy and needless harm if these rudimentary measures are not taken.

If you disagree about the transferability of the argument between the two situations can you tell me why? What makes the two technologies so different? Both block the doorways to avenues for producing harm. Both are sophisticated enough that it requires a nearly professional dedication to unlock. Both provide a measurable and significant increase in security for a community.

[skeaker]

The argument is not transferable because breaking into someone's house is sure to do more harm than the unspecified hypothetical harm that a "script kiddie" could do with ChatGPT, and that bypassing a door lock requires some degree of skill whereas a ChatGPT jailbreak requires you to google a prompt and copypaste it. A physical lock on a door offers a great deal more security than the limp solution that current AI safety provides, and it solves a much more pressing problem than "stopping trolls."

If your hypothetical involved a combination lock and the combination was on a sticky note that anyone could read at any time it might be more apt, but even then the harms done by breaking the security aren't the same. I'm not convinced a typical user of ChatGPT can do significant harm, the harms from LLMs are more from mass generated spam content which currently has no safeguards at all.