[trapexit]

LLMs are not a security barrier. LLMs cannot be a security barrier. They cannot form part of a security barrier. You must place the security barrier between the LLM and the backend systems, the same as you would place it between your web or mobile app and your backend systems. Assume that if the LLM agent can use a service, the human interacting with the agent can also call that service with arbitrary parameters.

The tools you're providing to your LLM agent must never have privileges greater than those you intend to afford to the user who is prompting / interacting with the agent.

You want to use an LLM to make a customer service bot? Sure, you can do that. But that bot MUST NOT UNDER ANY CIRCUMSTANCES be allowed to perform any action you wouldn't let the customer do himself. If it can read your CRM, you need to scope that access to exactly the same access you'd be willing to give the customer directly. Can it cancel orders? That tool must not be able to cancel any order you wouldn't let the customer cancel himself through your app or website.

Don't treat an LLM as if it could replace a human customer service agent, or a human researcher, or a human underwriter, or a human manager. Never make the mistake of believing that the LLM, with any level of clever prompt engineering or attempts at input sanitization, will be "good enough" at not getting fooled. If you trust it with the keys to the kingdom, in the same way that you'd trust a human with those keys, it's a matter of when—not if—you're going to get pwn3d.

Of course, holding this principle, if your autonomous agent can access the web, you must assume that literally anyone on the internet can call any of that agent's tools with arbitrary parameters.

[brabel]

This should be obvious to anyone who has ever developed an AI application. How are these companies deploying LLMs that have access to their full CRM Database and can just email that to anyone who asks nicely?! It truly is the 90's again.

[nicce]

Companies should think LLM just as an user interface, which is operating with the backend; the same principles apply. But the problem is that even today with traditional user interfaces, some companies will forget that the intended user interface is not the only part which should be secured.