[andyjohnson0]

"The neural network had learned to distinguish cloudy days from sunny days, instead of distinguishing camouflaged tanks from empty forest."

Reminds me of a story and ex-colleague once told me. He'd been working on a project to use neural nets to detect overcrowding on subway platforms using CCTV feeds. The NN worked with the training images but failed when run on live images. After a while they realised that training images showing crowding had been taken in a different season to those showing uncrowded platforms, and the NN was actually detecting the different type of clothes worn by the waiting passengers.

[rhsimplex]

This is exactly the kind of thing we designed this tool to prevent =)