[jerf]

Moving the problem around instead of fixing it is all that LLMs have as an option, which is why I believe they will in the end not be capable of doing most of what we're asking them to do. (But the next generation that uses them as a part, instead of trying to make the language center of the brain function as the whole brain, probably will.)

Fundamentally, for LLMs, everything is in-band. There is no way to signal out-of-band. They've got some ways of trying to indicate to the LLM in-band that this particular bit of content is out-of-band, but all they can really do is raise weights on that content. There's no way to say to an LLM this is rigidly, 100% out of band communication. Anyone who has worked in computer security for any length of time has been exposed to the extreme difficulty of securing things for which everything is "in band" communication. It isn't quite impossible necessarily, but when one sets out to secure an all-in-band system one is starting out a lot closer to "impossible" than I'm normally comfortable with. And that's for code that we write and humans understand, not billions of little floating point numbers.