[wxnx]

I think the evidence is pretty much in on that -- namely, yes, if your data is too small, a reasonably large neural net (a.k.a. basically any computer vision model from the last 3-4 years) is perfectly capable of memorizing the training images.

The relative success of attacks on nets to extract their training data support that this happens in practice too.

Generalization performance as it stands now always has to be evaluated empirically.