[maxkwallace]

This is a special case of a general class of vulnerabilities in AI models, where an adversary can cause undesired output from the model by constructing input data not represented in the training set. However, it is legit much more concerning than, e.g. the issue of image classification models mis-identifying well-constructed noise as "panda".

This is currently a research frontier for AI so us non-experts likely won't be able to say a ton about it.

I thought this was a good talk on the issue: https://www.youtube.com/watch?v=SS9DMr4VkbY