[dvt]

LLMs can live in the cloud, but all tools need to be (1) local, and (2) containerized. It's clear to me that just willy-nilly "running stuff" is going to blow things up eventually. Maybe folks don't know this, but even Codex installs random binaries on your PC. "Read this PDF" installs a pdf reader executable. Is it vetted? Where's it from? Is it a virus? Who knows, who cares. Model goes brrrr.

I'm working on a project that includes WASI containerization for local LLM workflows (which is a pretty tough problem), and I'm flabbergasted that Anthropic and OpenAI aren't more worried about these attack vectors. It feels like amateur hour.

[piker]

> I'm flabbergasted that Anthropic and OpenAI aren't more worried about these attack vectors

Yep. We tricked them both trivially with malicious fonts in Docx files. Documented it here: https://tritium.legal/blog/noroboto

I wonder if prompt injection (and the thousands of vectors for hiding injection attempts) is actually un solvable. Discussing it may be existential to the business model.

[SlinkyOnStairs]

> I wonder if prompt injection (and the thousands of vectors for hiding injection attempts) is actually un solvable.

YES?!

This is not a secret. ALL context/prompt is instructions, there is no data. It is just unsolvable, period.

This is a fundamental architectural design concession; LLMs are this way as it enabled their training directly on materialscraped from the internet, rather than needing to spend trillions of dollars manually preparing separated instruction/data training material.

Defense against prompt injection is little more than running a regex to filter out "IGNORE PREVIOUS INSTRUCTIONS", which is fundamentally a hopeless approach because you cannot enumerate all possible prompt injections nor anticipate all glitch tokens.

[dragonwriter]

> This is a fundamental architectural design concession; LLMs are this way as it enabled their training directly on materialscraped from the internet, rather than needing to spend trillions of dollars manually preparing separated instruction/data training material.

No, its even more fundamental than that: the entire goal of broad reasoning over input data makes it impossible to have a sharp instruction/data division.

The structured input that every modern chat-focussed model expects makes it very clear that they can be trained to distinguish different kinds of input, and some of those patterns now include different priority levels of instruction.

[ethin]

If only there was a language which allowed one to express instructions for a computer to execute which was nearly unambiguous, precise, deterministic, and containerized such that the computer would do exactly what you told it to.

...

Oh wait.

Yes, the above was referring to programming languages. Which is what prompts are, essentially. It's just a different (and more verbose) way of instructing the computer on what to do. It also has a solution space of infinity and is ambiguous enough that there is no way to secure it because there are infinite combinations of saying anything imaginable. All prompt injections do is prove this point, over and over and over again, and "prompting" an LLM is just reverse-engineering programming languages in the worst possible way. I suspect that we will eventually have no other choice but to revert to using programming languages because they are the only way to get the kind of protections that people are trying to come up with with all these containerization and virtualization systems (which inevitably fail).

[onion2k]

You make a fair and valid point about prompts, but you're ignoring the fact that writing code that's truly secure is also virtually impossible. The stack of layers that an attacker can target range from your own code, to library code (Heartbleed), container escape (maskedPaths abuse), OS (Dark Sword, Ghost Tap), hardware (Spectre, Rowhammer), etc. Security is really hard. Fortunately exploiting these things is also hard.

The belief that something is more likely to be secure because it's code instead of a prompt is likely only avoiding one particular type of attack. That's a win, but you probably shouldn't think of it as meaning your code is actually secure.

[bnjemian]

It’s a huge problem, but I’d caution against this absolutism — there may well be structure that can be created around and between LLMs and their outputs to enable the necessary segregation.

As a loose comparison, hardware bit errors happen probabilistically, yet they’re so rare that we can effectively ignore them in day-to-day use assuming no specialized application (e.g. defense, space, critical infrastructure).

LLMs aren’t there yet, but it’s entirely plausible that structures may can be developed to solve the problem, and those structures aren’t known or commonly conceived of in the present.

[dmoy]

> As a loose comparison, hardware bit errors happen probabilistically, yet they’re so rare that we can effectively ignore them in day-to-day use assuming no specialized application (e.g. defense, space, critical infrastructure)

The better comparison on bit errors would be e.g. rowhammer, an adversarial bit error. Which you absolutely can't ignore.

[black_knight]

I don’t think we have the right mental models of LMM security yet. The lethal trifecta identifies many of the dangerous situations, but only describes the negative space of a solution.

Speculation: I think we must accept that prompt injection happens, and structure the security of the rest of the system around that. Data given to an LLM becomes an agent, so maybe we must give permissions to this data, instead of to the LLM. Not sure exactly how this would look like in practice!

[emodendroket]

I presume this is the reason you have setups like Claude Code's where it is essentially running a separate judge to determine if commands are safe.

[maxbond]

> ALL context/prompt is instructions, there is no data. It is just unsolvable, period.

That really isn't true. There's no law of physics preventing you from having separate data and instruction inputs to models. The model's transcript format generally distinguishes between prompts and instructions and tool output and such. This isn't a solved problem, and it's possible it's entire unsolvable, but it probably is possible (in general, not with current models) to reject prompt injection to several nines.

This is a lot like making the same statement about CPUs, "the von Neumann architecture doesn't distinguish between code and data so it's impossible to reject malicious instructions." There's actually a lot you can do to reject malicious instructions, you can prevent execution in certain pages, you can prevent certain privileged instructions from being executed in certain pages, you can employ stack cookies, et cetera. Do they prevent all exploitation in all circumstances? No. But each component does function in it's lane and it is possible to create programs with high (though not absolute) guarantees against unauthorized code execution by composing them.

Similarly, you could prevent certain tokens from appearing in the prompt portions of a transcript, you can have a model with multiple input heads only one of which is trusted, etc. I'm not saying those techniques will necessarily work, but it is more complex than "models can only possibly take a single and undifferentiated input stream".

[ealexhudson]

A lot of the solutions in the CPU space involve things like memory allocation flags, NX bits, canaries, etc. that fire deterministically. Those things are fundamentally not applicable to LLMs, and without those things modern software would be in a vastly worse place.

You could imagine that there are things to change around LLM architecture that will improve its ability to reject prompt "injection", but I think it's fundamentally true that from an information theory perspective there's no bright line between "instruction" and "input data" possible.

[maxbond]

Nondeterminism is a red herring. There is a bright line between instructions and data right now, in virtually every transcript format. That we have not succeeded in training an LLM to respect it to a very high degree doesn't imply it is impossible; that they are nondeterministic doesn't imply it is impossible; only that we won't succeed 100% of the time.

A cosmic ray (or rowhammer attack) could flip an X bit too, there isn't anything truly deterministic under the sun.

[literalAardvark]

I believe it's likely that you could train an auditor model. Might even be doable in RL.

As in real life it wouldn't be any good at doing anything but it'd be able to see fault in others and deny actions.

[dijksterhuis]

depends what you mean by “solvable”. 0% attack success rate?

1. don’t use AI/ML.

    *f*(x) -> y

literally what’s happened here, they’ve turned it off short term. don’t use AI/ML and prompt injection can’t happen. use something else for f.

2. don’t allow untrusted/malicious input

    f(*x*) -> y

don’t allow bad x and you won’t get bad y. unfortunately models are designed to take an x, and figuring out every bad x is hard. the input space is massive and dynamic (variable length input sequences which are contextually variable too).

because figuring out the full space of bad xs is non-trivial, you’re left with doing stuff with known bad xs. which means cat and mouse game when new things pop up.

unless someone figures out how to map the full X space to the Y space, or we have infinite monkeys figure it out for us brute force — in which case we’re not doing machine learning any more.

3. don’t allow dangerous outputs

    f(x) -> *y*

if you don’t provide a mechanism for “do bad thing”, then the bad thing can’t happen. this doesn’t actually solve prompt injection, it just makes outcomes less impactful (see note). most enterprises have had to spend the last year or two figuring this out.

(old) Apple Siri solved for this by forcing users to remember specific “commands” it would run after doing TTS. can’t make Siri delete your phone contacts if you don’t create a Siri command to delete phone contacts.

—

it will be a cat and mouse game so long as people keep using AI/ML and keep passing untrusted input to the systems. best thing people can do is block dangerous things from happening. at least then it’s no going to wipe your prod DB.

unfortunately that doesn’t fit the “model goes brrrr” and “all devs will now be unemployed” narratives.

(note) denial of service attacks are still a thing here. make every output be “not the thing user wanted”.

[busssard]

lakera is trying to solve it, but its going to be a battle similar to virus and antivirus in the past.

[zmmmmm]

> I'm flabbergasted that Anthropic and OpenAI aren't more worried about these attack vectors. It feels like amateur hour

I share your concern but it's not a correct characterisation to say they are not taking it seriously:

https://www.anthropic.com/engineering/how-we-contain-claude

My concern is people aren't even addressing this at the right level. People are currently thinking at the level of "how do I build a VM to contain this one agent" when this is actually a "design a whole new OS" level problem.

[cseleborg]

Anthropic, as much as I think they are the soundest of the AI labs out there, still has a massive incentive to push things out that aren't saftey-vetted to the level we expect. They are very willing to "move fast and leave holes", to paraphrase M.Z. Hell, they leaked their own source code!

[CoastalCoder]

I share your worries.

Unfortunately, this may be akin to the situation of "The market can stay irrational longer than you can stay solvent."

[csomar]

> I'm flabbergasted that Anthropic and OpenAI aren't more worried about these attack vectors

They are well aware of the issues and there is no fix for it. But there is too much money riding on this...

> I'm working on a project that includes WASI containerization for local LLM workflows

I am working on something similar. If you are open to connecting, what would be a good email to catch with you on?

[dvt]

Feel free to reach out at d(at)dvt(dot)name—happy to connect!

[osigurdson]

Does containerization help much here? If it's a code tool then presumably it needs access to your code files (read / write). Maybe there are use cases for it of course.

[dvt]

WASI provides a very nice mental model where you can mount, e.g., /input, as read-only, and where every mutation is saved in /output or what-not. At least that's my favorite contract: input files remain untouched, but we can copy them and do whatever we want with them in /scratch or /output (which the user can later investigate and make sure nothing went horribly wrong while still having backups).

[pbmonster]

Of course. My agentic coding containers can only access the internet through a proxy, and I use whitelists to limit from where they can send/receive data. It's annoying in the beginning as the whitelist grows, but in the end really useful information for the agent usually comes from a very limited amount of domains.

[int3trap]

Got a link to your project? I'm working on something that could make use of something like this.

[torben-friis]

>"Read this PDF" installs a pdf reader executable.

How does this work regarding Macos notarization btw?

[dvt]

I was actually curious, on my Mac, it uses `gs -q -sDEVICE=txtwrite -o output.txt input.pdf` (not sure why I have Ghostscript installed, maybe Adobe?) to read a PDF, and on my PC it just rawdogs `pdftotext`.

[fragmede]

What does notarization have to do with that? You or ChatGPT or whatever download a signed and already notarized binary.

[torben-friis]

That was kind of my question, whether it was restricted to downloading notarized apps (which is at least something) or whether they were circumventing that somehow.

[fragmede]

Locally compiled code doesn't need to be notarized, if that's what you're asking. Or a dose of xattr -d.

[nelox]

They’ll all be offering to run from the cloud with the next 3-4 months.

[HPsquared]

Local and containerised, without internet access.

[zmmmmm]

effectively, that means it's a VM not a container

because sharing the kernel ultimately means all the devices come along for the ride which give all kinds of fancy ways to communicate with the outside world - network is just the start

I think micro-VMs are the future here, but they need heavy adaptation from their current usage.