|
|
|
|
|
|
by unholiness
11 days ago
|
|
|
No mention of KV cache, one of the biggest reasons not to switch models mid-stream. Once you pay, say, 50k input and 50k output Opus 4.8 tokens, you don't pay token cost for cache reads of the 100k context while it builds. Switch to another model, you'll start with the cost of 100k input tokens on the smaller model to get that context loaded and its unique KV cache set. The post may have some real insight here, where this 100k working context is actually better than trying to summarize it's findings into a plan. It's also right that the handoff is a perfect time to edit the context (removing planning instructions). But it doesn't mention it's a trade-off: the plan is smaller, so it's a cheaper "on-boarding" of the next model. Seems quite plausible that this is worth it for 1-off tasks. If it's right, this is basically "plans are useless, planning is everything" for LLMs. My problem is, I think the plans are useful. I want to review and edit them. I want them to give context for the upcoming code review (even if humans aren't reviewing). LLMs are notoriously bad at explaining why they're doing something in the moment. Humans are notoriously bad at accepting there's no reason why. I think the humans have it right here, and to bridge this gap want my PRs, my docs, and my comments teeming with reasons why. Plans help with that. |
|
|
> Opus reads base.py, signing.py, the test file (twenty cards of gray), then writes its plan and leaves. And what's the first thing Flash does with that beautiful document? It re-reads base.py and the test file, because a plan is not a file and you cannot edit prose. The gray reads just keep stacking, first at Opus prices, then again at Flash prices. There is no version of this where a second reader is the cost optimization.