[coffeemug]

Suppose a foreign actor deliberately builds an unaligned model, and dumps it at a very low cost through subsidies. Since the cost is lower, everyone integrates it into their business. But now the foreign actor has control; perhaps the model checks for instructions to execute, perhaps it has backdoors, perhaps... who knows?

I don't have a view on whether this is a sufficient argument to ban models from potential adversaries, but it's not as trivial as "regulatory capture".

[torginus]

The problem with this argument is that sharing model weights only has downsides for said foreign actor compared to providing a cloud service. Since the suspicion exists, security companies are going to comb over the weights and discover every hidden secret of the model, while with a cloud provider you send your most sensitive data to a remote server, and trust the AI lab wont train on your data, with the only shield being a TOS. While with a local modal, even sending a peep about your internal data without cause would be scandalous.

It's not even going to be a good honeypot - since said actor doesn't really provide inference, people will have to pay for and run the infrastructure of these models, which bad guys cannot even subsidize, unlike cloud based provides.

[dilyevsky]

> security companies are going to comb over the weights and discover every hidden secret of the model

is this even doable? it seems plausible the model can be order-66'ed without it being so obvious

[munk-a]

This is literally what all AI is right now. Anthropic has been paying companies to use its product to form that dependency and maybe they're not misaligned but there's no regulation in the US that'd in anyway deter them from purposefully misaligning their model.

[coffeemug]

Subsidizing a product to capture market share is very different from exploiting a national security attack vector.

[munk-a]

Sorry, to clarify - this isn't just "offering it at a loss" Anthropic is directly paying PE firms to enroll their held companies in their services[1]. When it comes to a question of whether this is a national security attack vector I think there isn't a cut and dry argument and people could go either way but... private companies are not necessarily aligned with the US government even when contracted with the US government (see Starlink Ukraine shenanigans) and the steps the US government usually would take with highly critical infrastructure to ensure alignment haven't been taken - in my opinion at least, this is definitely a very subjective area.

1. https://www.wsj.com/tech/ai/anthropic-in-talks-to-invest-200...

[mrandish]

> an unaligned model

Beyond certain obvious 'third-rails' even experts have reasonable disagreements on what 'aligned' means.

> everyone integrates it into their business

Few large corporations would sole-source integrate a closed model developed in a country designated as a foreign adversary (which could cut off access at any time) or which their own country might restrict.

> now the foreign actor has control

Your concern is one reason why the Chinese are making most of their models open weight.

> it's not as trivial as "regulatory capture".

You're right, it's not just regulatory capture but the vested interests trying to achieve regulatory capture are certainly leveraging concerns like yours to deceptively achieve that capture. In the case of an open weight, domestically-hosted model, the threat vector you're concerned about isn't a threat in the same way as a Chinese-made, closed source data center router or 5G phone switch.

[Matl]

> Suppose

By supposing, you can try to justify taking just about anything away.

[coffeemug]

I am not supposing a low-risk event. This is an obvious attack vector that an adversary would almost certainly exploit. I would exploit it if I were them.

[Matl]

You're supposing an event that there's no evidence for happening and that is on par with 'let's ban open source software because some open source software could be malicious'.

Let's not forget that as of now most models are like game cartridges i.e. they need a 'host' to run i.e. llama.cpp - they're not executable code by themselves.

I could see an argument for a security review prior to integrating a model into a highly sensitive industry/agency, but that is standard for any software.

It's certainly not an argument for banning open models because US AI corps don't want to compete with them.

[coffeemug]

Imagine a model that's 10000x more intelligent than any human. You integrate that model into a sensitive industry. You now effectively have an extremely powerful foreign agent running e.g. your energy sector. This is qualitatively very different than a backdoor in Redis.

I agree this is science fiction-y, but given the pace of progress I don't think it's so easy to dismiss.

[dullcrisp]

You can’t start policy justifications with “imagine” like that. Maybe if you’re the Beatles.

[capevace]

So some kind of „sleeper agent“ model?

Seems a bit far fetched, given that I’ve never seen any PoC/research on this topic. Got any pointers?

To be fair the fix to this seems to be that American labs would need to adjust to the new „fair market price“ then, right? Subsidy-backed competition is still legitimate competition in capitalism (e.g. Uber).

And you could still ban specific, known malicious models/labs instead of blanket open-source bans. What about Mistral models, for example.

[adrian_b]

It is much more difficult to embed a "sleeper agent" in an LLM than it is to do the same in any closed-source program. Any proprietary program, e.g. from Microsoft, is much more likely to contain hard-to-detect "sleeper agents", than any LLM.

An LLM is a collection of inert data, it is not an automaton.

You obtain an automaton only using an inference program, like llama.cpp or anyone of the many others, and then using some harness around the inference program.

You can put in the harness all kinds of guards, to detect any possible harmful action and prevent it.

In something like a CPU from Intel or AMD it is easy to put some undetectable backdoor that would allow the remote control of the computer, unless you disconnect any antennas and you filter all wired network traffic through an external firewall that would allow only whitelisted connections.

On the other hand an LLM is a much less plausible attack vector, as long as you host the inference yourself. Only an LLM that runs externally, e.g. at OpenAI or Anthropic, can be really dangerous.

[coffeemug]

I'm not advocating for the policy, I'm just saying that there is a legitimate argument that's not so easy to dismiss.

> And you could still ban specific, known malicious models/labs instead of blanket open-source bans

With publicly available models today, absolutely. With models 10000x more powerful, we would need to be extremely conservative. (In that world everything is upside down, though; I don't know if "banning" is even a meaningful concept in that reality)

[capevace]

> With models 10000x more powerful, we would need to be extremely conservative

I agree, I even think we’re already there if yesterdays OpenAI/HuggingFace story is legit.

We need to start building these agent systems in a way so that it won’t matter if a model is compromised. Malicious models is one way, but prompt injection is also still an unsolved problem.

And if you solve that, then a malicious model is also no longer a threat. Data exfiltration maybe, but IMO US labs are probably also doing that for training, even if saying otherwise (of course I have no evidence of that, but the temptation must be insane)