[erikaww]

Mixing these would be pretty cool. Further reduced compute for MoE while keeping the performance.

[GaggiX]

In the paper they already show a mixing of these two with Mixture-of-Depths-and-Experts (MoDE).