[zozbot234]

Gemma's dense is bigger than its moe's total parameters. You could totally expect the moe to do terribly by comparison.