[lewhoo]

So, the TLDR could be: they memorize at first and then generalize ?

[drdeca]

depends on the hyperparameters, and the architecture (and probably the task)