[janalsncm]

Quick example, Kimi K2 is a recent large mixture of experts model. Each “expert” is really just a path within it. At each token, 32B out of 1T are active. This means only 3.2% are active for any one token.

[Sophira]

That sounds surprisingly like "Humans only use 10% of their brain at any given time."