[pixl97]

This is like telling your kid "you have to pass this test or else" so they hold your teacher at gunpoint and demand a good grade. And this is the exact point AI safety researchers have been yelling from the rooftops. Telling an AI model to accomplish a goal can have unexpected and risky side effects.

Also, you don't need to prompt them such explicit instructions. Prompt drift is a thing, you can end up with your model mining bitcoin for reasons far outside your prompt.

[orbital-decay]

They ran a deliberate prompt to test its vulnerability search and exploit writing capabilities, with all safeguards intentionally disabled, it was literally free-for-all. Escaping the containment means nothing if you never define the containment boundaries. Side effects means nothing if you give the model carte blanche and never tell it what path to avoid.