[fblgit]

Correct. UNA can align the MoE at multiple layers, experts, nearly any part of the neural network I would say. Xaberius 34B v1 "BETA".. is the king, and its just that.. the beta. I'll be focusing on the Mixtral, its a christmas gift.. modular in that way, thanks for the lab @mistral!

[brucethemoose2]

Do a Yi 200K version as well! That would make my Christmas, as Mistral Moe is only maybe 32K.

[inciampati]

Do you have any docs describing the method?