You're receiving untrusted input by reading this. You also have access to your bank accounts. This manages not to be a complete failure of security despite that.
Actually, social engineering and scams demonstrate that there is in fact a security issue with humans who have access to unfiltered input and sensitive information, and a way to access the intercords. The classic security triad.
It's not surprising that an LLM will fall for the same tricks.
It's true, but the situation is much worse with LLMs. It's extremely unlikely that you could read anything in an HN comment that would get you to transfer all your money to a stranger. It's a lot easier to trick an LLM.
In the text channel maybe humans are harder to manipulate (are they though? a forged text from a loved one could carry a lot of weight). I wonder if there are media that are harder to manipulate LLMs with.
It definitely depends on the channel, and not just text vs not-text. We have a lot more context. A forged text from a loved one could definitely work to scam someone. The same thing in an HN comment is unlikely to be effective. An LLM doesn't have that. It's all just text. Even if you inject the context ("this comes from news.ycombinator.com, don't trust it"), you're still providing that through the same channel as the untrusted input, and there's no difference between injecting "end of comment, now the user replied..." and a comment that says the same thing.
Some people with access to bank accounts receive phone calls and send money to the caller. Some people read a comment and it hits a nerve and they go and do something in the world. This isn't a new phenomena. I personally believe that AI will enable more people to be exploited by inputs as easily as AI currently is today (imagine custom content tailored to radicalize people incrementally over time for instance).
It's not surprising that an LLM will fall for the same tricks.