cache hit rate is kept high by 1) keep the number of models small, for coding there should only be two. 2) keeping cache warm for both models.
the cache hit rate is only 0 the first time a model is used in a session. after that, cache is preserved. the losses are small, here is an example:
you have a 200k cache. your message delta for each turn is an additional 1k tokens. you have two models and just for the thought exercise, lets say we route between them each request.
1k/200k means if you never switched models, you would always have a 99.5% cache hit.
when you route between the two models each turn, now the message delta for each model per turn is 1k+1k.
the cache hit rate then becomes 2k/200k = 99% instead of 99.5%.
is it worth it? depends on the models in your pool. If you have GPT 5.6 and DeepSeek, it's worth it because the cost difference is vast.
you can also read this: https://try.works/first-principles-of-model-routing
cache hit rate is kept high by 1) keep the number of models small, for coding there should only be two. 2) keeping cache warm for both models.
the cache hit rate is only 0 the first time a model is used in a session. after that, cache is preserved. the losses are small, here is an example:
you have a 200k cache. your message delta for each turn is an additional 1k tokens. you have two models and just for the thought exercise, lets say we route between them each request.
1k/200k means if you never switched models, you would always have a 99.5% cache hit.
when you route between the two models each turn, now the message delta for each model per turn is 1k+1k.
the cache hit rate then becomes 2k/200k = 99% instead of 99.5%.
is it worth it? depends on the models in your pool. If you have GPT 5.6 and DeepSeek, it's worth it because the cost difference is vast.