[riku_iki]

its moe, each expert tower can be branched from some smaller model.

[jychang]

That's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.