[p1esk]

22,000-wide output sourced by a 4096-wide embedding

You will want to use hierarchical outputs in this case. Take a look at Hinton's 'Knowledge Distillation' paper.

[oneshot908]

Sure, that's a nice approximation, and it will reduce performance somewhat IMO, it'd be nice to quantify how much, no?