[chaos_emergent]

> This is likely because LLMs are typically trained for only a single epoch over massive datasets, which is in contrast to the multi-hundred-epoch training regimes for which dropout was first introduced.

Wait, is this true? That seems like a wild statement to make, relatively unsubstantiated?

[typon]

No this is well known. Look for Table 2.2 in GPT3 paper.

[chaos_emergent]

Thank you, that was a wild thing to learn!