[rughouse]

It’s very similar to Mixture of Experts. But instead of routing tokens to multiple experts, you "deploy to a single expert which can be dynamically skipped"

[erikaww]

Mixing these would be pretty cool. Further reduced compute for MoE while keeping the performance.

[GaggiX]

In the paper they already show a mixing of these two with Mixture-of-Depths-and-Experts (MoDE).