[gwern]

SGD doesn't work on large Transformers, no. You need something like AdamW.

[The_rationalist]

Mish is generally superior to RadamW https://lessw.medium.com/meet-mish-new-state-of-the-art-ai-a...

[dron57]

That's cool, but Mish is an activation function while SGD and AdamW are optimizers. Apples and oranges.