[ronyfadel]

It still does a much better job at translation than llama 2 70b even, at 6.7b params

[two_in_one]

If it's MOE that may explain why it's faster and better...

[yumraj]

MOE?

[sarthaksrinivas]

Mixture of Experts Model - https://en.wikipedia.org/wiki/Mixture_of_experts