Dogpile was only a good idea while Search Engines were mostly trash.
You needed to search all of them to find something decent.
That's roughly analogous to today. Ignoring cost, you'd be way better off asking all the LLMs to solve a problem (like coding) where you can verify the answer.
So the question is, for things like that -> can a group of models perform better than frontier models, especially at a reasonable cost?
Fable is not a great value, so unless you're trying to find answers to Erdos questions, you can probably do better on cost.
You can probably typically ask 3 or 4 of the top Chinese models for an answer and get a response for the same Fable question... Given that Fable isn't that much better, it's not surprising you can do better for a large subset of problems.
LLMs fail in such bizzare, obscure methods (to the average observer) at times. Sometimes even simple questions ("who was that x person who was super famous I'm thinking of") type questions fail terribly.
The more vague and non committal and hand-wavey and subjective the field for AI to answer, the better the results (imo).
Our whole stack is radically open source — frontend and backend alike, Apache-2.0 licensed — and so is everything behind this benchmark. That is how a benchmark number earns trust: verifiability, not hype.
Mixture of Models, perhaps? tbf to OP, the setup they're proposing has also been recently evangelized by other "ai gateway" products (like OpenRouter, JusCode, Fireworks etc), so there's likely something useful here.