[franga2000]

If you pipe your emails to bash, I can also run code by sending you an email. How is this news?

You must never feed user input into a combined instruction and data stream. If the instructions and data can't be separated, that's a broken system and you need to limit its privileges to only the privileges of the user supplying the input.

[rafram]

> You must never feed user input into a combined instruction and data stream.

Well, I have some bad news about how LLMs work...

[franga2000]

That's my point exactly. The only acceptable way to feed user input into an LLM is if its capabilities are constrained to only what you'd give the author of the input. If an LLM reads emails, it should only have the ability to create and display output, nothing more.

[Terr_]

I like to quip that people need to imagine LLM security as at-least-as-bad as javascript code in someone else's web-browser: A determined person can make them emit whatever they want (as noted above) and also none of the data that went into them is reliably secret either.

As an analogy, it still needs some work through, since it doesn't adequately alarm people about the risks of covertly poisonous data even with an honest user.

[QuadmasterXLII]

And even that’s imperfect if you miss an integration with an externally visible effect- for example an agent with web search can exfiltrate info via visiting specific urls with that log visitors- I’ve POC’d this with claude in the browser, although I only got a few bits out since you need to get N pages ranked on google to exfiltrate log (N!) bits

[furyofantares]

I suspect people routinely paste urls to documentation into claude code, which it will fetch, or maybe tell it to do web searches. It has those tools built in.

[ryukoposting]

You're right, but if all it can do is create and display output, then how will any of the oligarchs betting the farm on LLMs actually fulfill the breathless promises they've made?