[deyiao]

"The MOE architecture uses 20 times the parameters, is this comparison fair? Can it be compared with a single model that also uses 20 times the parameters?"

[jiggawatts]

It has more parameters, but not all of them are used during inference. They compared models that use equal numbers of parameters.