There are tons of anecdotes i remember from AI classes about neural nets learning something other than what you expected them to learn from your training set.
For example, one story involved training a classifier to recognize an overhead image with tanks vs without. It turns out it ended up learning which days were sunny and which were overcast.
This sort of thing happens when training people from examples too: From the mundane cases in school, to the AA587 crash in Queens NYC.
Nice sleuthing! I don't even recall where I first heard this story myself. The paper was an interesting read and still largely applicable even now in this 3rd or 4th coming of NN's.
It's kind of confusing, but table 2 shows what percent of these adversarial images trained on one networked worked on another. It varies quite a bit, and many networks aren't similar enough to each other for it to work reliably. But there is definitely some degree of generalization.
For example, one story involved training a classifier to recognize an overhead image with tanks vs without. It turns out it ended up learning which days were sunny and which were overcast.
This sort of thing happens when training people from examples too: From the mundane cases in school, to the AA587 crash in Queens NYC.