Mixture of experts: Make each model world-class within a single domain.
If adding one more common-sense QnA makes the calculus-bot even slightly worse at caculus, don't do it.
The “mixture of experts” concept in LLMs is a way of training a single model, it’s not based on training many different models (although that was the idea when the term was originally coined).