[hwillis]

"some caching gains" is a pretty huge understatement- snipping something out of the middle of the window requires rebuilding the entire context. Thats a shitload of tokens.

Afaik messing with the context also pretty reliably degrades performance still. The model responses reference things that no longer exist to it and it becomes more chaotic.

The real usefulness of parallel or sub-agents is not that they run at the same time, its that they isolate noisy or self-contained context away from the main window.

[MaxMatti]

I still feel that during agentic workload sometimes it would be nice to have the model identify it is veering off the main track, send out a "keep the cached states and tell me which they are" command to the inference server, do the side thing (such as handling an error that plopped up that has not that much to do with the main task) and return back to the cached state with just a comment tacked at the end to say "oh and btw I fixed DNS" instead of having the DNS debugging inside the context window now. Maybe other harnesses just steer the models more towards using subagents for such tasks and my pi is misconfigured. I can use the tree feature, but having insight into what's cached would be nice there.

[jdiff]

You can structure your context window to minimize the amount of editing you do further back. You usually only need to edit and correct the most recent response. It's little different from forking the conversation at an earlier point, and nobody warns about that being a sneaking footgun. There is still a prefix to cache.