[twobitshifter]

There’s a few options.

People think Gemini 1.5 is Sparse Mixture of Experts. (SMoE)

Another One is self extend. https://arxiv.org/abs/2401.01325

This paper also refers back to other options like yarn, etc.