[lrvick]

It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics.

They simply thought "Running in a VM/Container is easier and probably fine".

And the next 1000 escapes will be for the same reason, because negligence is quick and thus more profitable.

[simonw]

Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered?

How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?

[lrvick]

Models started proving readily capable of autonomously finding 0days over a year ago. At that point (ideally long before) the responsible sandbox for research labs with new models highly capable of this sort of thing should be literally airgapped. Disconnect the rack physically from any access to the internet, and give it an offline mirror of all of PyPi, NPM, Wikipedia, whatever as needed. They can afford the disk space. They already mirror it all anyway as training data. That is the bare minimum. Personally I would rule out side channel attacks with a $40k Tempest spec rack for good measure. Can interact with it via pubsub over a fiber optic serial terminal.

Such things are pretty normal in biotech, fintech, and defense who must have 0days in their threat models. AI labs absolutely need to start being forced to operate this way by endless negligence lawsuits. Move fast and break things culture is not going to cut it anymore.

[skybrian]

It seems like caching the relevant portion of PyPI in advance and turning the network off shouldn't be beyond what OpenAI can do?

[simonw]

I agree. I hope they learn that lesson and run future evals like this without pulling from external packaging repositories at all.

With my PSF board member hat on I'd like it if a whole bunch of other companies that make extensive automated use of PyPI (Microsoft and GitHub Actions for example) would do a better job of maintaining their own shared local caches.

[dist-epoch]

If you play with the ChatGPT web app sandbox, you'll see that they download PyPI packages from an OpenAI mirror/cache, and not from the Internet.

I know because they had a problem, and one package which was on PyPI failed to download for unknown reasons (possible size, 250 MB)

[simonw]

I just tried this prompt in ChatGPT:

  Show me all environment variables that
  start with PIP_ or UV_ or CAAS_ARTIFACTORY_

I got back a bunch of values like this:

  CAAS_ARTIFACTORY_PYPI_REGISTRY=packages.applied-caas-gateway1.internal.api.openai.org/artifactory/api/pypi/pypi-

That looks like https://docs.jfrog.com/artifactory/docs/remote-repositories

And from that documentation this does act as a caching proxy. The first time a package is loaded it's fetched from PyPI but subsequent fetches should be from the Artifactory cache, assuming it's shared across many different containers.

So yeah, I was wrong in this comment https://news.ycombinator.com/item?id=49015639#49024814 - they're caching already.

[paxys]

From reading all these threads there's clearly a large number of software developers on HN who can easily set up complex infrastructure and make it provably 100% bug and exploit free. Weird that they aren't all billionaires from selling these skills though.

[lrvick]

Unplugging the internet is easy. Running high risk code offline in highly tamper evident ways is hard, but plenty of teams like mine specialize in this sort of thing. (See: https://distrust.co and https://caution.co)

In our audits we regularly see billions of dollars in value at major companies at risk of theft by any anon paying attention that wants it, and we also absolutely know how to fix it and we maintain a lot of open source tools to accelerate doing so.

Some simply have no interest in any fixes that are not legally required or might have any short term impact on team velocity.

Survivors bias is a hell of a drug.

[jquery]

Being a billionaire has very little to do with engineering talent.

[customguy]

> How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?

You download the "website" and make it available locally. That's trivial.

[customguy]

guys if you need help just ask

I'm not up to hip with the speed because I think Python is very gross, but maybe to get a feel for it you could start with https://pypi.org/project/pypioffline/

I don't understand most of these words but I think if you paid me a lot of cash I could find someone who could help me make sense of it: https://techbeatly.com/offline-pypi-server-disconnected-envi...

Do OpenAI employees get paid? I'm honestly totally clueless about artificial telegents if you couldn't tell, but it's a company, right? Or is it more a community effort where everybody contributes when they can but they mostly have second jobs or even still go to school? In that case I say let's do a kickstarter so they can keep on being smart cutting edge high tech developers who aren't either a.) lying on the level of spam from Nigeria b.) doing their job at the level of someone falling for spam from Nigeria.

[abernard1]

> It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in.

It is, in fact, possible.

Do you not remember 2022-2023 when the air was singing with the sounds of young engineers in San Francisco elated they were creating an AI god?

And now a couple years later we're supposed to pretend they're competent?

Sorry. I've been in the tech industry a long time. There's a lot of incompetence. The margins provided by software allow a level of endemic incompetence that would simply not be possible in other industries.

[sailingparrot]

> they also clearly failed to contract specialists like myself to advise them on how to airgap software properly

Why would they want to airgap it though? They are trying to evaluate the model capabilities, alignment, potency etc. A model which will not run in an airgapped environment in prod.

So if you run your evals in airgapped environment, sure, the model doesn't bother breaking out of it's isolation and doesn't attack HF. But you have no idea what will happen once you release it in prod with internet connection, so are you in any way better off?

I would much rather have this happen while there is a single instance of the model running in a fairly well monitored environment, than when it's processing thousands of requests per second for real users, some with dubious motives, some with credentials right there on their laptop, some using it inside government facilities etc.

[lrvick]

You can simulate the internet in an airgapped environment for the tests and services you want it to interact with if you have enough disk space, and, they absolutely do.

[sailingparrot]

Sure, I'm not saying it's an untractable problem. But it's not as dumb as "how come a team of engineers paid millions haven't even heard of airgapping".

Creating a fake internet-like environment good enough to trick an advanced model that is very good at finding intricate flaws is not a simple job. Especially given that models can decide to behave differently once they suspect they might be under evaluation [1], which a fake internet would absolutely give away.

[1]: https://www.anthropic.com/research/alignment-faking

[lrvick]

I am not saying responsible research is easy, but I am saying that at their budget and scale there are no excuses to cut major corners on safety when the stakes and potential for very bad outcomes for others are this high.

[anematode]

For security??

Why else would they purposely lower cyber refusals in that case.

[sailingparrot]

> For security??

Really? By doing that you increase your security during eval and drastically lower your security at prod time, where internet is accessible, and where you are running much much much more requests in parallel, making it much harder to spot the one thats going rogue, in all kind of critical environment on potentially risky requests.

> Why else would they purposely lower cyber refusals in that case.

Because safety guardrails sometime fail, classifier misclassify, or can be inadvertently turned off by a bad PR etc. You can also imagine a more intelligent model working to go around guardrails by decomposing its actions into smaller ones that appear non-threatening to the safety classifier which does not have the entire context.

If you are going to deploy the system with internet access, you better be certain that you know what the worst case scenario WITH internet access looks like.

[tenuousemphasis]

Let me get this straight... Your solution to the AI box experiment is to never put the AI in a box at all?

[UltraSane]

Actually from what I read they did a very half assed job isolating the LLM from the internet. It would NOT be hard to do better.