[ml_thoughts]

Another more modern and well-documented example of this would seem to occur in a 2015 write-up of the "Right Whale" competition in Kaggle: http://felixlaumon.github.io/2015/01/08/kaggle-right-whale.h...

Contrary to this author's claims, despite using data augmentation and a fancy modern CNN, a neural network trained to identify whales hit a local optimum where it looked at patterns in waves on the water to identify the whale instead of distinctive markings on the whale's body.

I don't buy the "this isn't a problem in real world applications" argument being made in this article.

[vilhelm_s]

He says that his first attempt at whale recognition looked at waves instead of whales, but

> This naive approach yielded a validation score of just ~5.8 (logloss, lower the better) which was barely better than a random guess.

which is different from the tank story. For the tanks, the neural network appeared to perform well, but was actually not looking at the tanks. Here, it never performed well, and when debugging why not he found that it was not looking at the whales.

[Iv]

> I don't buy the "this isn't a problem in real world applications" argument being made in this article.

Me neither. Especially considering that this story was already alive before the latest deep learning advances. It is totally believable.

And even with a modern CNN approach, you would expect a model to be able to learn a sunny/cloudy categorization much easier than the nationality of a tank.

This story was repeated by professionals for ages because it is totally believable.