[jswny]

Unfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction.

For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact

[brookst]

Compacting at all is a mistake. With 1m context window there is no reason for a single task to require compaction.

Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after.

It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude) doing the compaction.

[wgjordan]

> With 1m context window there is no reason for a single task to require compaction

Only if money is no object. Cache reads are cheap (10% of uncached input costs) but definitely not free, and cached reads dominate session costs at long context lengths. A prompt at 20k context with $0.01 in cached reads would cost $0.40 in cached reads at 800k context, that quickly adds up for long sessions.

[brookst]

I’m not following the implication that these economics argue for jsing compaction instead of just clearing context?

[wgjordan]

Your 'with 1m context window' implied that some manually-curated task 'chunks' would overflow smaller context windows (eg 200-400k tokens). If you're instead curating chunks small enough to avoid getting burned on long-context cache-read costs, you're not using a 1m context window at all. At that point, compaction is a convenience over manually chunking and clearing context for fine-grained bits of work.

I think the stronger claim is: there is no reason for a single task to require a 1m context window.

[mrtesthah]

Most models’ reasoning abilities drops off significantly between the 256K-1M token ranges of the context window. There’s too much stuff to “pay attention to” at once.

[hedgehog]

I auto compact around 200k tokens both due to this and because the cached read cost really escalates when sessions have more tokens than that (too short and you pay a lot in per-compact re-reading of state)

[jeeeb]

Agree. Another consideration is that input token costs are effectively quadratic with the number of conversation turns. The minute you get into a long conversation you can see the costs shoot up.

You can get far more gas out of even a $20 plan of you’re careful to break things up into relatively small discrete steps, clear context regularly and give the model plenty of information to work with.

My workflow for bigger features is to write out a plan document and then proceed in smaller implementation steps, reviewing as I go. If I find something odd, I ask the agent why and often that leads to discovering a new dimension to the problem, which in turn is an opportunity to adjust the approach.

[kccqzy]

You just haven’t worked on tasks that are complicated enough. Occasionally it took more than 1M tokens just to come up with a plausible plan.

Personally I find using /rewind judiciously is better than using /compact. The latter essentially gives you no control of what details to discard, but the former at least has coarse-grained control.

[brookst]

Oh my goodness. I’ve used over 1B tokens on a single feature. I’m running at about 25B tokens/month right now.

My whole point was that by planning in advance you can shard the work into manageable sections with clear beginnings and outputs with acceptance croteria, and never compact, or even use more than a few hundfed thousand tokens in context.

It’s all hierarchical. Looking at an eval feature building right now, it’s 20ish build plans, each with zero to five or so /clear moments.

But maybe that’s the key thing… I don’t iteratively prompt ad hoc software writing. I do iterate on requirements, but if those are solid enough there is no “now write this function, now write that module”.

[irthomasthomas]

Have you ever looked at how much performance drops as context grows? The difference in intelligence between 100k and 1M is huge, like opus drops to haiku level performance, or worse. For that reason I try to keep under 200k. That feels about the upper bound for tasks requiring accuracy.

[tony_cannistra]

This is the way.

[weiliddat]

One interesting harness thing I saw Cursor do is to give the model access to the entire thread.

Even if it doesn't fit in the context window, the model can search through past turns and sanity check if something doesn't seem to be going right, or be prompted to follow an early message, "when starting on a new item, review the first message for how we should approach this"

[semiquaver]

Unlike other model/harness pairs, codex+gpt also passes an opaque encrypted artifact speculated to be an embedding representing the conversation back to the successor generation which is “denser” or at least higher fidelity than summarized text.

[perching_aix]

That's wild to read considering Codex's compaction to me feels like the model becoming demented. Super aggressive, throws away almost everything.

[jedisct1]

This is not an issue with Claude, but with Claude Code.

With swival.dev you can use long sessions without ever doing any manual compaction or reset, even with Claude models.

[what-the-grump]

Pretty much, I stopped using Claude for that reason, I shouldn't have to keep track of the context of the model all the time. And god forbid I don't, the amount of time I've lost to Claude just sending it and then lying is not worth it when 5.5 will just do stuff, compact 2-3 times in the task and still not go insane.

This whole 1 million context window is a lie after 300k Claude degrades to unusable, and compact doesnt help. I've had Codex tabs open for weeks, I have to regularly restart Claude.