[try-working]

OpenAI is doing this so that you can't preserve cache when routing between for example GPT 5.6 and DeepSeek V4, or other providers. I'm building a router that easily reaches >99% cache hit rate in Pi when routing between GPT 5.4, Kimi, DeepSeek etc. Working on a big update that's coming out in a week or so.

[confusus]

Awesome! Any place I can follow the project?

Also, how to achieve these levels when switching mid conv? Don’t you effectively need to read the tokens per model switch?

[try-working]

the repo is here and you can also find my twitter in my profile: https://github.com/try-works/role-model

you can also read this: https://try.works/first-principles-of-model-routing

cache hit rate is kept high by 1) keep the number of models small, for coding there should only be two. 2) keeping cache warm for both models.

the cache hit rate is only 0 the first time a model is used in a session. after that, cache is preserved. the losses are small, here is an example:

you have a 200k cache. your message delta for each turn is an additional 1k tokens. you have two models and just for the thought exercise, lets say we route between them each request.

1k/200k means if you never switched models, you would always have a 99.5% cache hit.

when you route between the two models each turn, now the message delta for each model per turn is 1k+1k.

the cache hit rate then becomes 2k/200k = 99% instead of 99.5%.

is it worth it? depends on the models in your pool. If you have GPT 5.6 and DeepSeek, it's worth it because the cost difference is vast.