[int_19h]

Yes. A mixture of experts is a single model that activates different routes though the same weights, with the route possibly changing literally on every token. It's not experts as in a bunch of standalone models that are good at specific high-level tasks.

[xnzakg]

On the other hand the standalone models would all need "basic" knowledge, so I'm assuming this would be much less disk space-efficient.