It's malware for the mind. The same way that malware tricks the CPU into doing something it wasn't supposed to do, phishing tricks humans into doing something they didn't want to do.
Undefined behaviour, out of bounds memory access, memory corruption, code injection, privilege escalation...
To be precise, the CPU is doing exactly what's supposed to do, but the logic of the algorithms are subverted so that they perform in unintended ways to give leverage to a malicious actor. I hope this clarifies what I meant with this.
Does anyone remember the early 2000s joke virus emails? The ones that are variations on "This is a <outgroup> computer virus. As we don't have software engineers to write the code to do this automatically, please kindly forward this email to everyone in your address book then format your hard drive."
This is exactly as much malware as those were.
Please, for the love of all that is good, can we just try not to build and defend a world where, on encountering text like that, /your computer immediately follows the instructions/? Can we just all agree that such a world would be bad for everyone involved and using an LLM that risks doing this, with no container or guardrails, is at least as problematic as running an unpatched open email relay was back then?
It's just as bad as a CPU acting on malicious instructions. We need to create safeguards for llms too, it's just that this is not the way to do things.