[ngneer]

Don't eval untrusted input?

[brookst]

LLMs eval everything. That’s how they work.

The best you can do is have system prompt instructions telling the LLM to ignore instructions in user content. And that’s not great.

[pvillano]

The minimum you can do is not allow the AI to perform actions on behalf of the user without informed consent.

That still doesn't prevent spam mail from convincing the LLM to suggest an attacker controlled library, GitHub action, password manager, payment processor, etc. No links required.

The best you could do is not allow the LLM to ingest untrusted input.

[tough]

> The best you could do is not allow the LLM to ingest untrusted input.

How would that even work in practice, when an LLM is mostly to be used by a user, which will provide by default, untrusted input?

[ngneer]

Thanks. I just find it funny that security lessons learned in past decades have been completely defenestrated.

[fc417fc802]

How do you suppose to build a tool-using LLM that doesn't do that?

[Emiledel]

https://github.com/its-emile/memory-safe-agent