[rrmm]

There are tons of anecdotes i remember from AI classes about neural nets learning something other than what you expected them to learn from your training set.

For example, one story involved training a classifier to recognize an overhead image with tanks vs without. It turns out it ended up learning which days were sunny and which were overcast.

This sort of thing happens when training people from examples too: From the mundane cases in school, to the AA587 crash in Queens NYC.

[cbr]

> ended up learning which days were sunny and which were overcast.

The earliest source I can find for this is https://neil.fraser.name/writing/tank/ but it says it "might be apocryphal".

[cbr]

Actually, there's also the 1992 paper "What Artificial Experts Can and Cannot Do" [1] which gives this anecdote and calls it a "legend".

[1] http://www.jefftk.com/dreyfus92.pdf

[rrmm]

Nice sleuthing! I don't even recall where I first heard this story myself. The paper was an interesting read and still largely applicable even now in this 3rd or 4th coming of NN's.

[cbr]

Expanded this into a post: http://www.jefftk.com/p/detecting-tanks