[NitpickLawyer]

> Fundamentally, LLMs are gullible.

I'd say that the fundamental problem is mixing command & data channels. If you remember the early days of dial-up, you could disconnect anyone from the internet by sending them a ping with a ATH0 command as payload. That got eventually solved, but it was fun for a while.

We need LLMs to be "gullible" as you say, and follow commands. We don't need them to follow commands from data. ATM most implementations use the same channel (i.e. text) for both. Once that is solved, these kinds of problems will go away. It's unclear now how this will be solved, tho...

[ryoshu]

This is a fundamental problem with these architectures. It's like having a SQL database with no way of handling prepared statements. I have yet to see a solution offered outside of rewriting queries, but that's a whack-a-mole problem.

[amelius]

Maybe simply turn every token input t into a tensor of shape 2x1 and use t[0] for the original input and set t[1] to either 0 or 1 depending on whether it is a command or data. Then train the thing and punish it when it responds to data.