[brookst]

I mean it's in the API reference: https://platform.openai.com/docs/guides/chat/introduction

Applications should not use user input for the system role. It's still not a firewall, but it's substantially better than the completion model from GPT3.

There was also a blog post / article / quite somewhere from OpenAI talking about how RL for GPT4 made it treat the system role as more immutable than was true in 3.5, but I'm not finding it in a quick search.

As the technology matures, we'll see security improvements as well. That's kind of the story of tech, right? SQL is doing pretty well despite having a similar problem with instructions versus user data.

I won't hang my hat on LLMs ever being perfect, but nor will I assert they are fundamentally broken and unfixable in this area. It is a very very young technology.