[Austiiiiii]

"How to prompt the model not to leak sensitive data" is not the right discussion to be having. It's a probability model, which means that every conceivable behavior is available in the confines of its code. There is no way to prevent an LLM with access to private information from divulging that information, or from attempting to sabotage systems it has access to. The only solution is to lock every LLM query in the entire stack behind the same deterministic role-based access controls that determine resources available to the current user.

I wish I could say I'm shocked a tech company architected internal systems with a built-in backend RBAC bypass like this, but with the degree to which they've marketed LLM-based solutions (on a subscription model that benefits them directly) as a wholesale replacement for deterministic code, it's no surprise they've become addicted to their own drug.

[smaudet]

"The only solution is to lock every LLM query in the entire stack behind the same deterministic role-based access controls that determine resources available to the current user."

Exactly. The sooner people stop trying to replace code with LLMs, the better. The technology is fundamentally untrustworthy, and given that we do not understand it, impossible to secure.

Only extremely simple code audited by multiple human authors, with actual proof of functionality (not just testing) can be considered secure.

[jsmith45]

Yeah, an agent should run with permissions no greater than that of the user on whose behalf it is executing, and ideally with less permissions. This is the scenario that is easier to fix, simply give the agent an API token with rights no greater than the user it is acting on behalf of. This could be a literal token for their account, or a limit-rights-to field or whatever, multiple possible approaches.

The harder problem is outside actors trying to prompt inject to get the agent to do something the user has rights to do but which the user doesn't want to happen. That is the hard scenario to fix, due to the nature of LLMs.

[catoc]

Exactly!

Attempting to handle prompt injections by prompting the model (not to leak sensitive data), is like attempting to stop a fire by burning the area around it

[politician]

My guy, that's actually how fires are stopped.

https://en.wikipedia.org/wiki/Firebreak

https://en.wikipedia.org/wiki/Controlled_burn

[catoc]

Haha, nice. TIL.

So all we need is ‘controlled prompting’ to handle prompt injections :-)

[NateEag]

It's maybe closer to putting a sign saying "The door is locked" on an unlocked bank vault.

It does nothing to improve security, and if someone manages to get inside and see the sign (i.e. "extract the prompt"), it gives them a strong hint there's interesting stuff behind this door.