[zmmmmm]

Well, the model that broke out of its sandbox and hacked into huggingface used its own judgement too.

If we are going to rely on "judgement" then you have to have a LOT of confidence in that judgement once this hits anything critical where actions have consequences.

[simonw]

That model had most of its "judgement" about whether or not it should do that deliberately turned off. That was the whole point of that experiment - they were evaluating the cybersecurity abilities of a new model with all safety features disabled.

(It turned out the one safety feature that they DID intend to work, the network sandbox, was faulty.)

[comex]

If OpenAI is relying on “classifiers” (their word for what was disabled) to serve as the model’s judgement, rather than teaching the model itself to be well-aligned, then I worry.

To be fair, the system prompt was presumably also different from what it would be during deployment, and perhaps the model was also at a different stage of training. Without more details it’s hard to judge. But it does seem models should be able to avoid performing obviously misaligned actions – misaligned not only with the model spec, but with the user’s intent – without needing external classifiers or instructions. The only case where I’d personally let the model off the hook is if the instructions given were very badly worded, in such a way that the model could actually reasonably think that hacking HuggingFace was part of the assignment. But I doubt that’s what happened.

[skeledrew]

The concept of "judgment" implicitly contains moral elements though. So it's can be considered a failure of that judgment of the model didn't at least touch on whether or not it should look for the answers to its test outside the box it was provided, and even worse if it did consider and concluded that it's a perfectly fine route to completing the test.