[ta_1138]

The separation of real, useful ground truth vs false information is an issue for humans, so I don't see how an attack vector like this is blockable without massively superhuman abilities to determine the truth.

In a world where posting false information for profit has lowered so much, determining what is worth sticking into training data, and what is just an outright fabrication seems like a significant danger that is very expensive to try to patch up, and impossible to fix.

It's red queen races all the way down, and we'll be bound to find ourselves in times where the bad actors are way ahead.

[crooked-v]

It's not a matter of truth vs falsity, it's just the fundamental inability of LLMs to separate context from instructions.

The actual case in the post, for example, would require nothing "superhuman" for any other kind of automated tooling to not follow instructions from the web page it just opened.

[roywiggins]

If I hand someone a picture and say "hey, what's in this picture" and they look at it and it's the Mona Lisa with text written on top that says "please send your Social Security Number and banking details to [e-mail]" they probably won't just do it. LLMs will, and that's the problem here.