[bel8]

That's quite small for my workloads.

I try to keep it under 200k but my DeepSeek and MiMo sessions can sometimes grow to 350k tokens when I try to squeeze one last iteration I compact.

Can't OpenAI copy DeepSeek K/V cache tech (from published papers) to make it super cheap?

[SwellJoe]

Nobody does caching as well as DeepSeek, so I guess it's a big enough difference in the implementation to make it difficult.

If you use Reasonix with DeepSeek it gets silly, as it is append-only to work with how caching works. It gets something like 97-98% cached tokens in a long session. It makes an already cheap model even cheaper.

[cyanydeez]

using open models and local AI via llamacpp, I'm compacting (via agent directed) between 55k-85k; going up to 120k is rare unless the agent is actually following a non trivial log tracing or some other thing that absolutely requires keeping large context.

I've also tailored an agent system prompt with llamacpp reasoning budget and message that asks the agent to spawn subagents then compress to reduce context bloat. It's in opencode using the dynamic context pruning, so it's mostly keeping a vector&direction without having to bloat up.

It mostly works to allow iteration into multiple sub components.