[swid]

This is very short sighted, and ignores the lethal trifecta insight.

The LLM doesn’t need to know what it is actually doing (it might think it is searching the web, installing a dev tool, or sending observability data (like metrics), when it is actually sending your API keys to an attacker (maybe in addition to what it thinks it is doing to keep it in the dark).

There have been some very clever things done I’ve seen… even a human reading the transcript may be surprised anything bad happened.

[raw_anon_1111]

The LLM would never have access to any API keys to send to the attacker. You send text to the LLM along with the prompt and it sends back JSON. You then send the JSON to your traditionally coded API. It’s not like your API has a function “returnAPIKeys()”.

As far as the LLM call, you are just sending your users text to another function that calls the LLM and reading the response back from the LLM.

If it didn’t create JSON you expected, your traditionally coded API is going to fail.

I keep wondering how are developers using LLMs in production and not doing this simple design pattern

[bigbuppo]

Oh man, this made me do a quick search on github. Looks like I picked the wrong week to stop quoting Zucker Brothers films.