[rishabhjain1198]

In a MoE model with experts_per_token = 2 and each expert having 7B params, after picking the experts it should run as fast as the slowest 7B expert, not a comparable 14B model.

[nullc]

Only assuming it's able to hide the faster one in free parallelism.

[moffkalast]

My CPU trying its best to run inference: parallelwhat?