[rohaga]

> So this only switches models if the cache is cold

Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, and so switching to Deepseek will save over time.

Remember that Deepseek can be something like 30x cheaper even cold.

There is a more fine-grained view of caching here, because each model can have a different amount of the prefix already cached as well (since we were switching a bunch) so even if the cache is somewhat warm it may make sense to switch.

[brandall10]

Seems like it might be more advantageous to just adjust reasoning effort to retain cache. Maybe in some exceptional cases where there will be a ton more inference to solve the problem, but going significantly dumber in that case seems counterintuitive.

I can really only see the utility of things like spawning subagents to a lower tier model from another provider, and that's something harnesses can already handle (ie. give model specs for certain delegation roles).

[benjiro29]

Seems like it might be more advantageous to just adjust reasoning effort to retain cache.

Some agents when you alter the reasoning level, partially or complete wipe the cache. Never assume that changing reasoning is a no-impact change. I do not have the link but a French youtuber did a ton of research on this.

[ValentineC]

Claude's cache breaks with a reasoning effort adjustment:

https://platform.claude.com/docs/en/build-with-claude/effort...

[rohaga]

I agree that adjusting the reasoning effort to retain cache is a huge thing! But even doing that automatically is currently a challenge for people to figure out and do well, and costs mental energy when perhaps it doesn't need to. For example, there is GPT-5.6-Sol low, med, high, xhigh, max, and lots of "rules of thumb" that people develop on which one to use when.

[verdverm]

caching is per model, it does not transfer between them

[brandall10]

Updated to clarify reasoning effort.

[ignoramous]

> the next task is SO HARD that it doesn't make sense to not use frontier

How does tokenless quantify "so hard"?

> 1 turn to crack versus 100 turns for Deepseek to crack

Interesting definition for a "frontier". What is a "turn" here? Token count? Request count? Context-based?

I've found that MiniMax M3 (a smaller model at 295b) will code up better when DeepSeek v4 Pro (1.6t) will not (and vice versa).

[rohaga]

Highly encourage you to read the blog post (https://usetokenless.com/blog/building-tokenless). Essentially, we estimate the confidence of a specific model failing or succeeding on a specific task using our own foundation models.

A turn here is a tool call/user input, anything that causes the model to get some new input. We're working on adding Minimax M3 and other models. We think that people have some intuitions about which models are good when--we seek to quantify them scientifically.

[CuriouslyC]

It will inspire more confidence if you say your classifier. Saying "your own foundation model" (charitably) suggests marketing hyperbole. If you're fine tuning a LLM to do this, at best it's going to be worse than a frontier model with suitably tuned skills (hence it should just be something in harness, not a service), and I wouldn't refer to that as "your own foundation model."

[gbalduzzi]

An approach that works really well for me in opencode is to use DeepSeek flash for the code exploration sub agent, and a smarter model for the overall task.

Code exploration tends to burn a ton of tokens and do not require frontier knowledge. It then returns a compact overview of the findings to the better model, that can therefore perform its task with less tokens.

Anyone else has a similar approach?

[sdesol]

The solution that I am working on is designed to leverage cheap LLMs (think MiMo, GLM, etc.) to build cheap context to reduce tokens, but most importantly to keep the context clean.

https://github.com/gitsense/chat/tree/update-readme

The README and latest version is still being worked on but I am hoping to have the latest release in place soon.

[dahdum]

How would your classifier compare to the frontier model delegating to Deepseek via a MCP tool call itself? Doesn’t it have far more insight on the difficulty of the task then your router could? I’ve been doing this with Claude Code delegating to Codex and an Ollama Qwen model.

[judge2020]

Would it not also be more efficient to promot to delegate more tasks where possible to just less capable agents? This is basically what Cursor does (although most certainly in an effort to get more grok/composer usage)