[hyperbovine]

Generalization error is a number. You can debate the probability space over which it should be computed, but it exists in one dimension. Otherwise statements like "generalize well" make no sense. Anyways, I'm not aware of any optical illusion that can make the brain confuse a house cat with a door knob. Yet it is apparently possible to make any one of a number of neural nets do so, using the same technique and with only subtle, imperceptible changes to the input. So they cannot possibly be learning any sort of intrinsic representation of these objects. I'm guilty of being facetious in using the word "just", since of course deep nets (and all other serious ML algorithms I know of) are able to generalize to an extent. What's not clear to me this represents some paradigm shift in AI, as is often claimed, or if it's simply the consequence of fitting a hugely overparameterized function approximator to a web-scale amount of training data.