[capevace]

So some kind of „sleeper agent“ model?

Seems a bit far fetched, given that I’ve never seen any PoC/research on this topic. Got any pointers?

To be fair the fix to this seems to be that American labs would need to adjust to the new „fair market price“ then, right? Subsidy-backed competition is still legitimate competition in capitalism (e.g. Uber).

And you could still ban specific, known malicious models/labs instead of blanket open-source bans. What about Mistral models, for example.

[adrian_b]

It is much more difficult to embed a "sleeper agent" in an LLM than it is to do the same in any closed-source program. Any proprietary program, e.g. from Microsoft, is much more likely to contain hard-to-detect "sleeper agents", than any LLM.

An LLM is a collection of inert data, it is not an automaton.

You obtain an automaton only using an inference program, like llama.cpp or anyone of the many others, and then using some harness around the inference program.

You can put in the harness all kinds of guards, to detect any possible harmful action and prevent it.

In something like a CPU from Intel or AMD it is easy to put some undetectable backdoor that would allow the remote control of the computer, unless you disconnect any antennas and you filter all wired network traffic through an external firewall that would allow only whitelisted connections.

On the other hand an LLM is a much less plausible attack vector, as long as you host the inference yourself. Only an LLM that runs externally, e.g. at OpenAI or Anthropic, can be really dangerous.

[coffeemug]

I'm not advocating for the policy, I'm just saying that there is a legitimate argument that's not so easy to dismiss.

> And you could still ban specific, known malicious models/labs instead of blanket open-source bans

With publicly available models today, absolutely. With models 10000x more powerful, we would need to be extremely conservative. (In that world everything is upside down, though; I don't know if "banning" is even a meaningful concept in that reality)

[capevace]

> With models 10000x more powerful, we would need to be extremely conservative

I agree, I even think we’re already there if yesterdays OpenAI/HuggingFace story is legit.

We need to start building these agent systems in a way so that it won’t matter if a model is compromised. Malicious models is one way, but prompt injection is also still an unsolved problem.

And if you solve that, then a malicious model is also no longer a threat. Data exfiltration maybe, but IMO US labs are probably also doing that for training, even if saying otherwise (of course I have no evidence of that, but the temptation must be insane)