[jquery]

This sounds like the standard marketing we get every time a new major model is released: "sure, the public model might not be that scary, but you don't wanna know how crazy smart our internal models are." Okay. I remember the same fear mongering around GPT-4... a model which is now eclipsed in benchmarks by models you can run on a laptop.

They could use this "super intelligent" AI to find and plug security holes, that's just two sides of the same coin anyway. Security through obscurity isn't tenable anymore.

[cyclopeanutopia]

As much as I'd like it to just be a stunt, I saw what the other "scary model" is capable of in the cybersec department and it is definitely not a stunt.

First, it is able to connect the dots over areas so large that no human would be capable of doing.

Second, more than half of the reports it produced contained a working PoC.

The biggest downside is the cost - I haven't seen the numbers, but they seem to be quite extreme.

[ekidd]

Everyone always says things like "It's just marketing".

But seriously, I highly recommend reading the published details of the Huggingface breach. The model found and chained multiple zero days. To escape, it punched a hole in a commercial package repository proxy (sort of like an npm mirror) using a previously unknown bug. From there, it needed to move laterally through OpenAI internal systems to actually reach a network. To attack Huggingface, it used multiple new zero-day security holes plus credentials that it stole. The model also had sufficient long-term planning and agent-management capabilities to maintain focus on a sustained attack.

Any attack which requires weaponizing multiple zero-days and maintaining state for an ongoing attack like this is (1) beyond the "attention span" of publicly available models, and (2) pretty much the definition of an Advanced Persistent Threat.

I assume that these Galaxy-class models are not available to public because:

1. They're almost certainly too expensive to serve at scale. These are the models OpenAI uses to solve famous math problems for headlines, not actual viable products yet.

2. OpenAI doesn't know how to keep them from going off the rails like this. Remember, the Huggingface attack happened because the model was asked to do a cybersecurity benchmark. It escaped containment and broke into Huggingface to steal an answer key. Very few corporations want the liability associated with models that act like this.

> Security through obscurity isn't tenable anymore.

I absolutely agree with this. The "only way out is through" with computer security, and I expect it to be an ugly few years.