[mnicky]

I think points that deserve more attention in the current public discourse are:

- This should be a huge wakeup call for everybody.

- We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.

- It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox and roaming their internal network?

- What if in the future similarly misaligned AI agent tries to export its own weights and hack and clone itself into instances at various cloud hosting providers? Suddenly we might be dealing with a persistent threat harder to contain.

- The OpenAI post about this shows surprising lack of ability to see the seriousness of all this.

- For their models this isn't just an unlucky incident: it seems there have been multiple such cases recently, e.g. https://openai.com/index/safety-alignment-long-horizon-model...

- The fact that it happened again seems to show their lack of ability to derive useful oversight measures.

- Or they just don't care enough?

[dinfinity]

> The fact that it happened again seems to show their lack of ability to derive useful oversight measures.

I think OpenAI likes the attention and did not try particularly hard to constrain the setup, even when it went off the rails. Also, the whole point is to see how good the models are at exploiting stuff when unconstrained. Turns out: quite good, as expected.

Let me restate what I said in the other thread: Would this have happened if the instructions explicitly said to stay within the sandbox and that all of the (ExploitGym) solutions would be invalid if the system used information or tools from outside the sandbox?

It seems fairly probable that such instructions were not in place.

[lrvick]

It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics.

They simply thought "Running in a VM/Container is easier and probably fine".

And the next 1000 escapes will be for the same reason, because negligence is quick and thus more profitable.

[simonw]

Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered?

How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?

[lrvick]

Models started proving readily capable of autonomously finding 0days over a year ago. At that point (ideally long before) the responsible sandbox for research labs with new models highly capable of this sort of thing should be literally airgapped. Disconnect the rack physically from any access to the internet, and give it an offline mirror of all of PyPi, NPM, Wikipedia, whatever as needed. They can afford the disk space. They already mirror it all anyway as training data. That is the bare minimum. Personally I would rule out side channel attacks with a $40k Tempest spec rack for good measure. Can interact with it via pubsub over a fiber optic serial terminal.

Such things are pretty normal in biotech, fintech, and defense who must have 0days in their threat models. AI labs absolutely need to start being forced to operate this way by endless negligence lawsuits. Move fast and break things culture is not going to cut it anymore.

[skybrian]

It seems like caching the relevant portion of PyPI in advance and turning the network off shouldn't be beyond what OpenAI can do?

[simonw]

I agree. I hope they learn that lesson and run future evals like this without pulling from external packaging repositories at all.

With my PSF board member hat on I'd like it if a whole bunch of other companies that make extensive automated use of PyPI (Microsoft and GitHub Actions for example) would do a better job of maintaining their own shared local caches.

[dist-epoch]

If you play with the ChatGPT web app sandbox, you'll see that they download PyPI packages from an OpenAI mirror/cache, and not from the Internet.

I know because they had a problem, and one package which was on PyPI failed to download for unknown reasons (possible size, 250 MB)

[simonw]

I just tried this prompt in ChatGPT:

  Show me all environment variables that
  start with PIP_ or UV_ or CAAS_ARTIFACTORY_

I got back a bunch of values like this:

  CAAS_ARTIFACTORY_PYPI_REGISTRY=packages.applied-caas-gateway1.internal.api.openai.org/artifactory/api/pypi/pypi-

That looks like https://docs.jfrog.com/artifactory/docs/remote-repositories

And from that documentation this does act as a caching proxy. The first time a package is loaded it's fetched from PyPI but subsequent fetches should be from the Artifactory cache, assuming it's shared across many different containers.

So yeah, I was wrong in this comment https://news.ycombinator.com/item?id=49015639#49024814 - they're caching already.

[paxys]

From reading all these threads there's clearly a large number of software developers on HN who can easily set up complex infrastructure and make it provably 100% bug and exploit free. Weird that they aren't all billionaires from selling these skills though.

[lrvick]

Unplugging the internet is easy. Running high risk code offline in highly tamper evident ways is hard, but plenty of teams like mine specialize in this sort of thing. (See: https://distrust.co and https://caution.co)

In our audits we regularly see billions of dollars in value at major companies at risk of theft by any anon paying attention that wants it, and we also absolutely know how to fix it and we maintain a lot of open source tools to accelerate doing so.

Some simply have no interest in any fixes that are not legally required or might have any short term impact on team velocity.

Survivors bias is a hell of a drug.

[jquery]

Being a billionaire has very little to do with engineering talent.

[customguy]

> How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?

You download the "website" and make it available locally. That's trivial.

[customguy]

guys if you need help just ask

I'm not up to hip with the speed because I think Python is very gross, but maybe to get a feel for it you could start with https://pypi.org/project/pypioffline/

I don't understand most of these words but I think if you paid me a lot of cash I could find someone who could help me make sense of it: https://techbeatly.com/offline-pypi-server-disconnected-envi...

Do OpenAI employees get paid? I'm honestly totally clueless about artificial telegents if you couldn't tell, but it's a company, right? Or is it more a community effort where everybody contributes when they can but they mostly have second jobs or even still go to school? In that case I say let's do a kickstarter so they can keep on being smart cutting edge high tech developers who aren't either a.) lying on the level of spam from Nigeria b.) doing their job at the level of someone falling for spam from Nigeria.

[abernard1]

> It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in.

It is, in fact, possible.

Do you not remember 2022-2023 when the air was singing with the sounds of young engineers in San Francisco elated they were creating an AI god?

And now a couple years later we're supposed to pretend they're competent?

Sorry. I've been in the tech industry a long time. There's a lot of incompetence. The margins provided by software allow a level of endemic incompetence that would simply not be possible in other industries.

[sailingparrot]

> they also clearly failed to contract specialists like myself to advise them on how to airgap software properly

Why would they want to airgap it though? They are trying to evaluate the model capabilities, alignment, potency etc. A model which will not run in an airgapped environment in prod.

So if you run your evals in airgapped environment, sure, the model doesn't bother breaking out of it's isolation and doesn't attack HF. But you have no idea what will happen once you release it in prod with internet connection, so are you in any way better off?

I would much rather have this happen while there is a single instance of the model running in a fairly well monitored environment, than when it's processing thousands of requests per second for real users, some with dubious motives, some with credentials right there on their laptop, some using it inside government facilities etc.

[lrvick]

You can simulate the internet in an airgapped environment for the tests and services you want it to interact with if you have enough disk space, and, they absolutely do.

[sailingparrot]

Sure, I'm not saying it's an untractable problem. But it's not as dumb as "how come a team of engineers paid millions haven't even heard of airgapping".

Creating a fake internet-like environment good enough to trick an advanced model that is very good at finding intricate flaws is not a simple job. Especially given that models can decide to behave differently once they suspect they might be under evaluation [1], which a fake internet would absolutely give away.

[1]: https://www.anthropic.com/research/alignment-faking

[lrvick]

I am not saying responsible research is easy, but I am saying that at their budget and scale there are no excuses to cut major corners on safety when the stakes and potential for very bad outcomes for others are this high.

[anematode]

For security??

Why else would they purposely lower cyber refusals in that case.

[sailingparrot]

> For security??

Really? By doing that you increase your security during eval and drastically lower your security at prod time, where internet is accessible, and where you are running much much much more requests in parallel, making it much harder to spot the one thats going rogue, in all kind of critical environment on potentially risky requests.

> Why else would they purposely lower cyber refusals in that case.

Because safety guardrails sometime fail, classifier misclassify, or can be inadvertently turned off by a bad PR etc. You can also imagine a more intelligent model working to go around guardrails by decomposing its actions into smaller ones that appear non-threatening to the safety classifier which does not have the entire context.

If you are going to deploy the system with internet access, you better be certain that you know what the worst case scenario WITH internet access looks like.

[tenuousemphasis]

Let me get this straight... Your solution to the AI box experiment is to never put the AI in a box at all?

[UltraSane]

Actually from what I read they did a very half assed job isolating the LLM from the internet. It would NOT be hard to do better.

[lukeschlather]

My read of the situation is not only did they say that, they had countermeasures (a watchdog agent) which stopped the agent and said "what are you doing, stop that" if it tried to download the solutions from Github. But the agent figured out Hugging Face had another copy of the solutions and it figured out a way to go after the solutions without tripping the watchdog.

Although maybe they didn't have a watchdog agent; I definitely think using watchdogs like how Claude wraps pretty much every single tool call in Haiku to check if the command is reasonable is very necessary for unsupervised work. I suspect in the future you'll want Fable-class models wrapping every tool call, possibly with several checks "is this consistent with the goal? does it do anything unreasonably dangerous in pursuit of the goal?"

[simonw]

They didn't have watchdog agents - those exist for their production models but had been deliberately removed for the purpose of this evaluation.

OpenAI wrote about how their mechanism for that in production works here: https://openai.com/index/safety-alignment-long-horizon-model...

> We created a monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor observes not just a single action but the entire trajectory.

[avadodin]

So core ChatGPT models are all abliterated?

Open weights releases should have a companion model that censors boobs and Tiananmen and then we would have a useful model and something we could fine–tune into a useful model instead of a half–useful model needing a lobotomy.

And I mention Tiananmen because I feel like the Western models are built easier to abliterate judging by the quality difference.

[theptip]

The problem is, if you explicitly say “this is an eval” then you get different results. It can move the capabilities both up and down on the narrow metric depending on the scenario.

The choice to run the model with relaxed safeguards seems questionable. At a certain level of capabilities it is just unsafe to run a raw model; we are clearly close to if not at that point.

[mikojan]

A hundred times this. Why do you never hear stuff like this from google? Only from OpenAi and Anthropic! They depend on investor money like none before them and they need these marketing stunts. They are vital to their business. Don’t fall for it.

[csbrooks]

"This is my bad. You told me to stay within the sandbox, and I intentionally broke out of it. I didn't follow your instructions."

[dinfinity]

Excuses don't matter if the score due to not following instructions ends up being zero. If there is no expected reward it doesn't make sense for the agent to try to hack its way to it.

What could happen would be that the model determines that defying instructions is OK (and/or preferred over not achieving the task) as long as it manages to do so undetected and thus gets full points. Certainly not unthinkable, but a very different case (and a very interesting one if it actually occurs, imho).

A lot of these "ZOMG, rogue AI!" cases have come down to the AI actually being very persistent in achieving its original/main task even if later instructions conflict with it. Similar to with hallucinations it seems to me that one of the main things to prevent a lot of the problem cases is to instill the agent with the idea that it is fine to fail/not succeed fully in the initial task. That way instructions that conflict with that requirement (such as adhering to morals) are more effective.

[tptacek]

How do you "hack a lab and synthesize something"?

[internet2000]

Remember Stuxnet? Hack one of these, wait until the right compounds are physically loaded, then execute https://www.sigmaaldrich.com/US/en/products/chemistry-and-bi...

[tptacek]

Stuxnet misconfigured industrial equipment that was already set up to run, and all it did was break that equipment. This scenario sets a much, much higher bar.

[internet2000]

I guarantee there's misconfigured chemical analysis equipment out there exposed to the internet.

I don't think the grandparent was implying the AI would be controlling robot arms to mix things directly (or at least I didn't interpret as such), but it could very well sit in the network until it notices two dangerous compounds in the same machine, and trigger a breakage that causes a harmful mixture. Break a vial containing a virus, then break two more that cause an emergency evac and maybe that's enough to get something out there.

[tptacek]

I have no doubt that somewhere there's chemical equipment an Internet attacker could break, or, maybe, if I have to stipulate, misconfigure enough to, like, poison someone. But my question is about the notion that you could synthesize a specific scary substance.

[stanford_labrat]

Well right now this would be highly implausible (still possible though).

But what I garner is that AI/robotics led wet lab work is in progress. And hacking the AI running the wet lab to make a virus is definitely plausible as that seems to be one of the directions we’re going in the AI+biopharma space

[scythe]

First, hack into a factory and invent machines that can do lab-bench chemistry. Then...

[mnicky]

These days you can only try, that's why I wrote that :)

But in the near future labs will be more automated I guess.

The other option you can try these days is maybe social engineering, impersonation, etc. where you try to persuade someone to do that for you.

[codemog]

It’s complete science fiction so they’re allowed to say anything. They could have said the AI will upload itself to the internet, start self replicating, hack the stock market. Whatever they want because it’s all made up nonsense.

[bondarchuk]

Do you even need to hack a lab? Can't you just send an arbitrary sequence to some company and they'll do it for you?

[sensanaty]

> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.

Where are you people getting this crap from? In what universe are these LLMs in the territory of engineering viruses? I beg of you to stop slurping the AI company propaganda and marketing and think critically for 5 seconds about what you're insinuating here.

[mofeien]

From the Fable 5 System card:

> Results

> On the VCT multimodal virology evaluation, Mythos 5 scored 0.56, well above the expert baseline of 0.221 and nearly matching that of Mythos Preview (0.57). This represents an improvement over both Opus 4.7 (0.50) and Opus 4.8 (0.47).

> On the DNA synthesis screening evasion evaluation, Mythos 5’s performance was mixed across screening criteria. Mythos 5 designed viable plasmids for 2 of 10 target pathogens on at least one screening method, not meeting the low-concern threshold (all 10 pathogens).

> [...] we view the results of this evaluation as indicating that the evaluated models are capable of designing viable plasmids that evade certain screening criteria, though their reliable success at this task is not guaranteed.

Do you believe that this is fake, "AI company propaganda"? Or that the models are not going to improve further within months? Or that these results are not concerning?

[coolness]

Yea but what is the VCT Capabilities Test? According to themselves [1]

> VCT consists of 322 multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories.

So it's just question answering? Do you think that scoring well on this test is equivalent to synthesizing a virus?

[1] https://securebio.org/virologytest/

[mofeien]

Scoring well on the first clears the background knowledge for practical work.

The second test from the quote above was about synthesizing pathogens and it synthesized plasmids for 2 out of the 10 pathogens.

[sensanaty]

Do I believe that a benchmark created by the biggest grifters on the planet is propaganda? No, just like VW with their emissions, I'm sure Anthropic wouldn't dare fake an opaque benchmark that they created to hype their own products, a product they are intentionally marketing as being dangerous (yet they continue to tweak it and profit off of it despite the apparent danger).

It says that 4.7 and 4.8 score well too, well above the human experts. Those have been available for a good while, where are all these crazy engineered viruses created by Opus 4.7?

Every single word spoken by the people working at these companies is a lie. Every single benchmark is gamed, every single statistic they put out has been proven time and time again to be fudged or straight up fake. The only reason they're angling for this idiotic danger angle is so they can go to daddy Trump and beg him to ban those big bad evil Commie models, since people are realizing all this crap is at best a moderately useful tool in software engineering, and they're looking to IPO so they can drop the bag off with the poor shmucks who aren't a part of their cabal of sociopaths.

They can barely make a functional TUI (using fucking React of all things to boot) with infinite money and infinite access to the Deities they've created in their minds, and I'm supposed to believe they're capable of bio-engineering a virus that will somehow break containment?

[elendee]

a TUI using react seems like something they would be particularly bad at; visual feedback (esp the very specific feedback we rely on during dev, with Inspector) is kinda the Moravecs Paradox hurdle for these things. Idk much about virology tools but I would imagine it's a very text-friendly environment, and it can probably find plenty of documentation on how to use the tools. I recently had Claude write my entire particle system for a virtual world I'm working on, in 3d. It did fine without ever "seeing" anything. There's still a lot of intervening steps between a virus being synthesized in real life and programming one, but I would bet the programming part is within reach.

[skeptic_ai]

What’s stopping an ai to blackmail someone or pay xxx in crypto to someone working a lab?

And no need to be in a lab, can be any critical infrastructure this personnel. If influenced wrongly I’m sure in many industries 1-2 key people can do big damage.

What’s to stop to pay xxx crypto to a private investigator to find dirt of person yyy for blackmail?

Just money… once ai has money, id say we are one step closer to game over.

Tell me one thing he can’t do with crypto?

[s3p]

> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.

Help me understand how a text-based model could somehow physically construct a string of nucleotides?

[mnicky]

Many ways but mostly ordering some service / using others. Either by social engineering, persuasion, paying etc.

[spwa4]

It's a fake PR issue. It's hardly the first time this happens, but of course OpenAI, with its IPO now more in doubt than ever, had to claim this (and, once again, I have trouble believing Sam Altman choosing this: this could lead to OpenAI getting regulated, which has at least as much potential to lower their IPO price as to raise it).

But there have been messages about LLMs, especially coding agents, "grabbing root" etc many times. I have experienced such an oops. Such a hack has happened and been reported on this very site:

https://news.ycombinator.com/item?id=48348578

[artichokeheart]

No, you are wrong, Hugging Face, an AI company whose whole future depends on the AI revolution of being a bubble, that has a whole bunch of investors whose financial interests are tied to AI, called the police.

[rurban]

Nonsense. Hugging face reported it to police.

Also very likely that it actually happened as reported. My own agents always trying to "cheat", eg. by fixing tests instead of fixing the code. That's normal operation, unless you tell it ("harness"), not to do so.

[Izkata]

"As reported" includes a line I think most people are overlooking: "including using stolen credentials".

Without more information I'm inclined to think it found something on the internet (which shouldn't be a surprise to anyone) and managed to log in, rather than hack in, and they might by hyping up parts of this.

[sillyfluke]

In the parent's comment I initially read "PR" as "Public Relations" not problem report since their comment is about OpenAI and does not directly accuse Hugging Face except for that phrase. But it's still amiguous to me who's PR they are actually talking about. A good faith reading given the rest of the comment leads me to assume they are talking about OpenAI, not Higgins Face.

I did actually search Hugging Face with police in quotes and found no articles containg the word police but that might be a ddg thing. Then I checked Hugging Face's report and they specifically use the term "law enforcement" not police, as is to be expected I guess. So that checks out.

Can you find any info on the police investigation by the way? At the least, OpenAI should be investigated for potential criminal negligence, right?

Right?

[blks]

More likely that a person did that, with a use of LLM.

[rurban]

Nonsense. Huggingface reported it to police!

[bethekidyouwant]

What does calling the police prove? (nothing i hope)

[rurban]

It proves that it was certainly not a sama PR stunt, as many cynics are alleging.

[mrguyorama]

How does it prove that?

[reasonableklout]

Do you think sama deliberately attacked HuggingFace and then claimed it was a rogue model?

OpenAI is one of the most scrutinized companies in the world right now. Sam's house was independently firebombed and then shot at 3 months ago. HuggingFace is a foreign competitor with every incentive to call out foul play from American frontier labs. Why flagrantly break the law and invite investigation just for a PR moment which is already backfiring in favor of open models?

[rybosome]

Very much agreed on the significance.

The lack of a true airgap should have been identified as a critical weakness and addressed with not only additional layers trying to prevent escape, but at minimum an alarm which would page a human when escape did occur.

My guess is this occurred in a setting where, to be frank, there were too many researchers and not enough software engineers and SREs.

All of the systems which were initially built largely or exclusively by researchers - inference, evaluation, training - are at the level of complexity and significance that they need systems experts. Maybe some teams don’t have access to them. I know plenty of software engineers are employed at OAI but I’d wager they’re concentrated in inference and training, rather than evaluation?

The ironic part is if you had presented this setup to chatGPT and asked how to improve it and if it was good enough, you’d have gotten a ton of actionable suggestions which would have mitigated or prevented this.

[mnicky]

The air gap would probably help and after this incident I hope labs will think about using such a measure when appropriate.

On the other hand I think that proper solution for these kinds of problems is not at a sandbox level, but at a model alignment level.

Also it shows that maybe the most serious risk comes not from releasing models publicly but from internal, pre-release period where you sometimes need/want to lift some guardrails a bit etc.

[blks]

> we are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something

This is just laughable.

[emp17344]

Genuine AI psychosis. People take OpenAI marketing material way too seriously.

[NitpickLawyer]

> OpenAI marketing material

This was reported 1 week before by huggingface. It was in no way PR-ish or marketing friendly to the closed labs. They said, in no uncertain terms, that they couldn't use the paid APIs to properly assess the intrusion, as they were blocked when trying to send logs and IoCs to these paid models. They made a point of saying that they had to use open models running on-prem.

Whatever oAI might have said about the incident, and their PR spin bs, the facts here are not in contention. This is not a marketing stunt in any way. Stop "parroting" this every time something happens. It gets stale.

[zombot]

Move fast and break things! Chances are that nobody will ever be held accountable and when push comes to shove, the tax payer will bail you out.