[vladf]

Alternatively, one could get rid of the memory used by optimizers entirely by switching to vanilla SGD.

I haven’t tried this on transformers and maybe that’s what breaks down here but in “classic” supervised settings I’ve found SGD with schedule tuning just as fast as Adam.

[gwern]

SGD doesn't work on large Transformers, no. You need something like AdamW.

[The_rationalist]

Mish is generally superior to RadamW https://lessw.medium.com/meet-mish-new-state-of-the-art-ai-a...

[dron57]

That's cool, but Mish is an activation function while SGD and AdamW are optimizers. Apples and oranges.