[buster]

The sentence "It is astoundingly easy to push an LLM over to collapse into ungrounded nonsense" makes me wonder.

How easy? What specific methods accomplish this? Are these methods fundamentally different from those that mislead humans?

How is this different from exploiting cognitive limitations in any reasoning system—whether a developing child's incomplete knowledge or an adult's reliance on heuristics?

How is it different from Fake News and adults taking Fake News for granted and replicating bullshit?

Research on misinformation psychology supports this parallel. According to https://www.sciencedirect.com/science/article/pii/S136466132...:

  "Poor truth discernment is linked to a lack of careful reasoning and relevant knowledge, as well as to the use of familiarity and source heuristics."

Perhaps human and LLM reasoning capabilities differ in mechanism but not in fundamental robustness against manipulation?

Maybe the only real difference is our long term experience and long term memory?