[joe_the_user]

I think this might also be due to the fact that the compute for neural nets and the complexity of the networks are still in their infancy.

Well, neural nets may be just starting out but I think one can they're approximation process are not complex. They are very complex in the sense of having many layers and many pseudo-neurons on each layers.

What's happening is that the networks are mapping images to high-dimensioned "feature space" and then drawing dividing line in the feature between matching and not-matching images. It is vastly complicated but heuristic process. Essentially, the division between image types are based on both meaningful and meaningless differences between the images. The example classified as "a boy holding a dog" (when it was a goat) and "a herd of giraffes in trees" (when it was goats that had climbed trees happened to have more random characteristics in common with the classification than their real qualities.

The thing is the method can be made relatively better but for absolute improvement, you'd want a way to not just have more approximation but to find a way to get rid of garbage approximation, garbage conclusions and so-forth. I suspect that would imply both different algorithms and a different training cycle.