While I'm here - I recently optimized our PR review skill to make better use of the KV cache.
Previously, it loaded up the diff, persona and prompt, and wrote those into the first message for each of 6 sub-agent reviewers. The prompt was templated with the persona name, so was slightly different for each reviewer.
My optimized version had a common first message with diff + prompt, and a script to run to atomically claim a persona. It also runs the first reviewer before the rest to warm the KV cache, and the agent doesn't launch the rest of the subagents until the first persona has been claimed (which means that the LLM is running, and therefore the KV cache is warm). The agent has a script it runs which does the waiting for it.
Agents 2-6 only run once the KV cache is warm, and because the intro message including the diff is shared, it's there in the cache. Only the persona file is different.
In my testing this brought down the cost of a review by ~half, though of course this depends on how big the diff is, how many agent are launched, and what model is used.
The layout, as well as all of the paragraphs are written by me manually, although I generally draft a completely broken English one very rapidly and ask it to paraphrase, I do apologize if the editorial pass made it harder to read, but it really isn't an LLM's opinion :)
I thought KV cache not being as important was obvious, as a big part of the benchmrks were focused on pricing, but in retrospect I can see how I could've made that more clear.
In the end cost is what matters most and as seen from the results the KV cache miss on the cheap model doesn't really matter as much.
What is there to talk about the KV Cache, they’re handing off to a different model, I thought that you can’t reuse KV cache between entirely different models?