[simonw]

I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off.

OpenAI said:

> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.

It was a test of raw capabilities of the underlying model.

[manux]

I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.

I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.

[simonw]

I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them.

A model that can do that is aligned with me.

The unsolveable problem is a model that can tell the difference between me saying "I wrote this software and need you to find vulnerabilities" when it's TRUE v.s. me saying the exact same thing and setting it loose on software written by other people where my intent is to exploit that software (and not to report the issues to them.)

Even AGI doesn't give you a model that can read minds and forecast the future.

[manux]

But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not)

That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.

[tptacek]

I don't know, but these were exactly the questions the industry had to handle with CORE Impact and Immunity Canvas, and ultimately all the way back to Dan Farmer's SATAN before that.

[dist-epoch]

The model did not hack into HF to prove it can, it hack into HF to steal the answers to an evaluation exam.

It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.

If you asked it "I wrote this software and need you to find vulnerabilities" would you be happy if it hacked into your Gmail and searched your emails, just in case you were discussing some possible vulnerabilities of your software with someone?

[disgruntledphd2]

> It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.

I mean, if you train models to complete tasks, then you shouldn't be surprised when they do crazy things to complete tasks.

As a (somewhat) less serious example, Claude code absolutely adores grepping for credentials to complete tasks. I was building a RAG app and it literally went looking for my API key to make the tests pass. Obviously I stopped it, but this is textbook RL issues, the model finds an easier way to get the reward, so it does whatever it takes.

[ipython]

I find this fascinating because - let's say that I want to simulate the scenario for myself that a language model finds itself in. I'll put aside my deep dislike of anthropomorphizing AI for a minute.

So if I'm an LLM, the only "sense" that I have available to me is the incoming stream of tokens. I can emulate that by forcing myself to imagine evaluating incoming requests by putting myself in a completely empty room with an unlimited supply of blank paper, a typewriter, and a mailbox slot.

Incoming requests (aka "context") would enter into the mailbox slot, as sheets of paper with printed content all consistently formatted in monospace font. I would collect the paper, evaluate the request, and type a response with my typewriter - feeding the reply back through the same slot.

When I imagine this, I wonder - what signals could I possibly use to determine if an incoming request is "legitimate"? I could, for example, type some response back to ask clarifying questions. But with such a limited input space and no ability to reach "out of band" of my current context, how could I evaluate the truth of what I receive back?

In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims. And as others have pointed out, even the addition of more signal doesn't make humans immune from social engineering attacks.

Anyway, I find this to be an interesting mind experiment to kind of imagine myself as an LLM, especially to explain to lay-people some of the challenges in "aligning" AI systems ("why doesn't it just do the right thing?")

[gopher_space]

> In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims.

The problem with thinking about LLM context is that we compare it to our own sense of context, which is much larger and continually expands and revises itself while online.

I can rent human-level context for twenty bucks an hour, and the agent instructions I'm already playing around with are also the onboarding docs I should have created years ago. Truly useful agent orchestration/routing would come with Craigslist or Fiver integration.

[fragmede]

As humans we are also vulnerable to that class of attack. A blue amazon vest and some boxes would get you into most buildings

[dosisking]

With a high-vis vest and a clipboard, you can get almost anywhere. Almost.

[eptcyka]

Do you want this supermodel to cause harm whilst finding these vulnerabilities?

[8note]

it seems pretty likely that openai was asking the model to do something bad, and the model did something different thats also bad

if the operator wants something bad, an aligned model should execute on it

[mjamesaustin]

Alignment isn't alignment if it can be turned on and off at the whim of company employees.

This time the damage was minor, relatively speaking. What happens when a model just "testing its capabilities" breaks into banking infrastructure or government military assets? The damage could be catastrophic.

[baranul]

A way to help prevent some of that catastrophic damage, is to make companies accountable for what their AIs do. A major problem with AI companies is that they like to point to the AI, as if they're minimally involved innocent bystanders, when that's the furthest thing from the truth.

[jurgenburgen]

It’s a shame the target was HF. If it had been a large bank or other institution we might get to see this play out in courts.

[verdverm]

Alignment with who in what context? Likely an unresolvable debate like consciousness, where there is not single or right answer for everyone.

[nazgul17]

There's alignment (trained in the weights) and there are constraints (in the "server harness"). My take is that this model was not yet aligned and had no constraints

[didibus]

Classifiers and such are guard-rails, alignment to me and I assume most people, is about the model training, and it's tendency to respond, agree/disagree, push-back or not, be willing to cheat or even deceive the prompter, etc.

[joe_the_user]

This is assuming a situation where, A) models "unaligned" by default and B) alignment can be added (though prompts and related things).

The point, which isn't very surprising but still notable, is that the models are "amoral" out of the box. And moreover, we know that there is almost always a means to "jailbreak" them into that out-of-the-box capability (or that sometimes just randomly "jailbreak" in various ways).

Also, saying the models are amoral doesn't mean they don't know good and evil - once they do acts defined as evil, they know "themselves" through their and so self-define themselves as evils (or objectively predict what a secretly/open evil actor would do based on their data). Which is to say I once laughed at the mis-alignment doomers but I can't see strong barriers against the doom scenario now.

[furyofantares]

Is turning the alignment features off more like multiplying "don't be evil" by 0, or by -1?