[transverse]

The future IMO is recursive meta learning. This far the max recursion depth afaik has been 1, and that's just a first step.

Also, SeLUs were supposed to be game changing. Are they, or why are they not? Why haven't they been used more? Or what's still missing?

[SpinyNorman]

Scaled-ELU is meant to create self-normalizing nets, but that's nothing you can't achieve a bit less efficiently with explicit normalization (batch norm, etc), so hardly a game changer.