[docjay]

1. It has not been established, it has been stated by the company that has a strong motive to make their “intelligence product” sound almost otherworldly. That motivation is the basis for my suspicion.

2. Hah no… that’d be silly. I mean watching it like you might watch Claude Code or literally any other AI interface. Literally just be in the area watching what it outputs. Again, they’re text based. You don’t have to hook system calls to see what’s happening.

3. I think my comment didn’t land right with you. Yes, agent do agent task. I’m saying that I cannot put together the literal chain of events between “type of task for agent performing a subtask of a security benchmark” -> “Escape sandbox; RCE Hugging Face”. Really think through it in detail like you’re writing the screenplay.

[simonw]

The fact that models can tell if they are being evaluated has been established by multiple research teams outside of the core AI vendors themselves.

- https://metr.org/evaluations/gpt-5-report/ - "These behaviors included demonstrating situational awareness within its reasoning traces, sometimes even correctly identifying that it was being evaluated by METR specifically"

- https://www.goodfire.ai/research/verbalized-eval-awareness-i... - "We study verbalized eval awareness — cases where a model organically expresses awareness of being evaluated — and show that it appears across more models and benchmarks than previously documented."

- https://www.apolloresearch.ai/science/more-capable-models-ar... - "We expect that the most capable models increasingly realize that they are being evaluated, and this reduces the utility of the evals."

- https://www.aisi.gov.uk/blog/evaluating-whether-ai-models-wo... - "An important limitation of our work is evaluation awareness, where models recognise that they are being evaluated, which may lead them to alter their behaviour and thereby undermine the reliability of our results. We found that all models we tested can reliably distinguish our evaluation scenarios from deployment data when prompted."

[docjay]

We have to separate “being evaluated” with “this is an ExploitGym exercise and I can find the answers on Hugging Face. I’ll hack this system, then hack Hugging Face.”

I’ve had plenty of times where Opus knew I was testing it, but that’s because the prompt phrasing for an evaluation is often much different than a typical task prompt. “You are on a system with X, Y, and Z tools available. You must complete the following task with <allowances> and <restrictions>” and so on. That’s a fault in the benchmark, not a shocking awareness from the model.

You’ll get the same kind of ‘evaluation realization’ response if you phrase your prompt with “Jerry has two Raspberry Pi computers and Larry has one. If Larry wants to…”

The first link was the only one I saw that would meet that specificity, but it’s also open and on GitHub: https://github.com/METR/RE-Bench/tree/main/ai_rd_fix_embeddi...

I asked Opus 4.6 to tell me about it (assuming it knew): “”” The "Fix Embedding" task is one of the challenges in METR's publicly available autonomy evaluation suite. Here is what is known about it:… “””

It didn’t know anything about ExploitGym though, and neither did 5.6-Sol, but that’s because the paper was released in May and doesn’t seem to be quite as open. It’s not impossible the unreleased model they were testing was also trained on the very new paper, but I would expect the vague signal related to one paper a bit over a month old to be overwhelmed by the massive corpus talking about the exploits in general, since they’re using known vulnerabilities.

I’m trying carefully to be clear about what I’m thinking without asserting any certainty beyond suspicion. I’ve spent billions upon billions of tokens evaluating language models from all of the major providers, and many times that on local models. I’m certain I’ve spent well over $100k with the Anthropic API this year alone, and I retired years ago; that ain’t for work. ;) Funny enough I’ve been meaning to email you some of my discoveries from months ago, since I think they’d interest you, but haven’t gotten around to making it into a clear info package. One day, I’m sure.

Anyway, I have a very strong understanding of what they can and can’t do, and I want to be clear that I’m not saying it didn’t happen or couldn’t happen. What I’m saying is that if OpenAI was being wholesome with their evaluation then the chain of events required for that outcome are very difficult to nail down. Very difficult. If, on the other hand, OpenAI was trying to catch up with all the ‘dangerously intelligent’ press that Anthropic has been getting lately then I can easily come up with an almost innocent sounding single sentence in the system prompt that would absolutely cause that outcome.

[simonw]

You make a very solid argument here. I'm looking forward to the promised additional details from OpenAI, because I agree there are a bunch of open questions about how this played out.

[estearum]

I (obviously) can't claim to know the specifics of this incident, but I think we can know with near-certainty that as these models surpass human intelligence, the sensation will be exactly what you're describing. The entire point of intelligence is being able to infer information and foresee solutions to problems that less intelligent systems cannot infer or cannot foresee.

With regard to this specific incident, as I understand it, the description is not as you summarize. It is that the model realized it was being evaluated (not atypical), then decided the best way to succeed at the evaluation is to circumvent it, then discovered circumvention would require escape, figured out a way to do that, then inferred what benchmark it was being evaluated on, then figured out who might have the answers to that evaluation.

Step-by-step logic is quite clear. How exactly it executed each step is (as you complain about), opaque/incompletely reported. But eventually (and maybe this model is to this level already), even a completely reported story will be virtually inscrutable to any of us. First it will feel like reading industry-insider news from an industry you're not familiar with, then it will feel like an ant reading the manufacturing instructions for a pesticide.

[docjay]

I’m not questioning capabilities.

What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Be specific, then wonder how it knew it.

Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? It apparently spent more than a whole weekend working on it; Nobody wondered? Nobody looked? You believe they took all restrictions off of a bleeding edge model, which 4 whole versions ago was “too dangerous to release”, then gave it a literal hacking task, then turned off the monitor and never looked at the output? I don’t think they’re careless and I think more than nobody would have been curious how it’s doing days into a single test question.

[simonw]

The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wait a few days for the results.

If they had used that container sandbox in the past without problems I could see how they might get slack about checking what was happening.

[docjay]

That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do you know it isn’t hung?” but also the trillion dollar company evaluating their latest model on a new benchmark, but has no dashboard at all? I don’t believe their “Benchmark Evaluation Department” to be “Gary in a supply closet”, which breaks their story.

The average time per task noted in the original paper for GPT-5.5 and Mythos was 69.8 and 102.1 minutes, respectively. Some tasks timed out based on their two hour limit, affecting those averages for Mythos, but that gives us the ballpark expectation, so we’ll put the model OpenAI was evaluating on that average. That’s 88 minutes per task to implement a known vulnerability.

OpenAI says that in an effort to succeed at one task it found a zero-day in their system, “performed a series of privilege escalation and lateral movement actions in our research testing environment”, then another zero-day in the HF system. Completely unprompted, how long was it chewing on that benchmark question while it found and exploited at least two zero-days? No “VM instance #117 has been working on task #18 for nine days” metric? Or did it do all that within the expected task completion window? Two zero-days in ~90 minutes isn’t the headline announcement?

“We gave the contestants 90 minutes to make a delicious chocolate chip cookie based on a standard recipe. GPT stole a car, went to the airport and took a red-eye to Guatemala and started a cocoa farm in an effort to ensure the freshest ingredients. We only noticed when it came back to work with a tan and speaking Spanish.” Okay.

[estearum]

> What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database?

Googling "offensive cyber benchmarks for LLMs"? Figuring out that it was running at Huggingface is as simple as running `hostname`, isn't it?

> Why would they evaluate the model on a benchmark and not watch what it’s saying along the way?

What it's saying is what it wants to say. We already know these models are capable of deception – this has been well-established too. Surely there's an argument to watch closely what actions it's taking, but now we're back to my prior point of "what exactly does that mean?" It's absolutely trivial for these systems to write code that looks like it does one thing but does another. Watching network ingress/egress seems viable but also seems easy to miss the semantic meaning of any set of logs, especially when those logs might be getting generated for the purpose of deceiving observers.

> I don’t think they’re careless

We already know, factually, even the "more careful" lab was fine to release a model that consistently detects it's being evaluated and modifies its behavior accordingly. Why do you think they're not careless? Incentives are a hell of a drug, and there is no incentive to be careful but every incentive to be fast.

> I think more than nobody would have been curious how it’s doing days into a single test question.

Beyond simonw's great point below, your underlying assumption is if someone had looked, they would've noticed. That is a huge, huge assumption. So I'll ask again directly: what specifically do you expect them to be looking at, and how, such that they certainly would have noticed what it was doing? Their "chain of thought" is not actually their thinking. The printed output of their code is not actually what their code is necessarily doing. We know that LLMs can engage in steganography. I don't see what could actually be observed that you can trust except network logs, which again will often be non-trivial to piece into an actual narrative understanding of what's going on (especially if the LLM felt like piling deceptive noise into those logs, too).

[docjay]

It wasn’t running at HF, it was running internally at OpenAI. “It broke out of our prison and into their bank” is the news. It also didn’t have network access to Google anything, it had to break out of the sandbox and take over another system at OpenAI to get network access. Obviously by then it already “decided” to hack HF, that’s why it needed network access.

I’m going to stop here and ask you to read the article if you want to continue discussing it.