<i>"The neural network had learned to distinguish cloudy days from sunny days, instead of distinguishing camouflaged tanks from empty forest."</i><p>Reminds me of a story and ex-colleague once told me. He'd been working on a project to use neural nets to detect overcrowding on subway platforms using CCTV feeds. The NN worked with the training images but failed when run on live images. After a while they realised that training images showing crowding had been taken in a different season to those showing uncrowded platforms, and the NN was actually detecting the different type of clothes worn by the waiting passengers.