[appplication]

Generally, posting a link-only reply without further elaboration comes across as a bit rude. Are you providing support for the above point? Refuting it? You felt compelled to comment, a few words to indicate what you’re actually trying to say would go a long way.

[noosphr]

>We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better.

[ForceBru]

Right, isn't double descent one of the reasons why modern Extremely Large Language Models work at all? I think I heard somewhere that basically all today's "smart" (reasoning, solving math problems, etc) LLMs are trained in the "double descent" territory (whatever this means, I'm not entirely sure).

[mxwsn]

No, there are more training tokens than parameters in LLMs. They are in the classical first descent setting.

[SiempreViernes]

No, double descent is a symptom of whatever it is that makes the deep models work at all. It's just the name for something you see happen when it works. The reason it works has something to do with how all those extra dimensions work as a regularisation term in the fit.

[smallerize]

Does this mean that if your model is "overfitting", the solution is to train for even more epochs?

[noosphr]

Maybe. Just means that the conventional wisdom was wrong and substantially over training can be a good thing. No one I knew at the time suspected that, including the people who wrote the paper.