[scionaura]

Insider take: routing will not be a (successful, durable) thing, at least not externally to model providers.

The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut into lower levels of abstraction.

E.g. you should consider speculative decoding to be one (very conservative) form of routing and note that you can’t implement that for the labs from the outside.

[CuriouslyC]

This arbitrage isn't durable either though, as it's just a race to the bottom with open research, and the long term destination of everything is custom models for different tasks, since it's going to become increasingly apparent that some areas of knowledge have anti-synergy as we push up the diminishing return curve.

[brunaxLorax]

100% agree. MoE is the perfect example: reducing FLOPs while keeping intelligence