[kmeisthax]

No. Mixture-of-Experts is for dynamically loading different sets of weights[0] per-token. The idea with MoE is that you can have more weights than fit in RAM without having to actually load everything in RAM.

My idea is more about creating recurrent paths in an otherwise all-forward network. The same number of weights would be loaded, we'd just be routing residuals differently.

[0] Usually multi-layer perceptron / linear portion weights - although maybe someone's tried attention head MOE?