[tosh]

I can only recommend to regularly measure how many tokens a harness+model combination uses for a certain task

There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment

Yesterday I ran 10 agentic tasks using GPT 5.6 Sol in an ubuntu 26.04 vm a couple of times with different harnesses and got vastly different token usage.

  +-------------+-----------+-----------+-----------+-----------+--------+
  | Harness     | API total | Input     | Cached    | Uncached  | Output |
  +-------------+-----------+-----------+-----------+-----------+--------+
  | smol        |   172,807 |   142,334 |     8,704 |   133,630 | 30,473 |
  | Pi          |   427,211 |   392,767 |   137,216 |   255,551 | 34,444 |
  | OpenCode    | 1,564,429 | 1,523,957 | 1,204,736 |   319,221 | 40,472 |
  | Codex       | 3,005,744 | 2,953,154 | 2,649,344 |   303,810 | 52,590 |
  | Hermes      | 3,856,611 | 3,808,231 | 3,167,232 |   640,999 | 48,380 |
  | Claude Code | 5,073,137 | 5,029,969 | 4,587,008 |   442,961 | 43,168 |
  +-------------+-----------+-----------+-----------+-----------+--------+

https://x.com/__tosh/status/2083593799872237680

I'm not surprised that Claude Code is not optimized for an OpenAI model but I was still quite shocked re how much of a difference the harness makes.

Disclaimer: I'm working on 'smol' which is a minimalist harness but it's really nothing special, just a minimal system prompt, no skills files, only tool is shell

Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.

[yojo]

Claude Code injects a ton of tools into the system prompt, including their “memory system” that’s like 10k+ tokens. Depending on your task shape, this can easily double your task cost (e.g. a low context-using job that takes many turns, like a monitoring loop).

You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.

[KronisLV]

> Claude Code injects a ton of tools into the system prompt

It's so unfortunate they don't let you use the subscription with other harnesses anymore - since even if I used OpenCode they'd still get a bunch of useful data from the API calls, meanwhile I could stretch their tier limits way further.

[kelnos]

The funny thing is that in some cases using a different harness with the subscription plan could actually be very good for Anthropic: e.g. if I were to use smol with Opus, it could use fewer tokens than CC for the same task. People on subscription plans burning fewer tokens is a good thing for Anthropic.

The only downside for Anthropic that I can see is that hitting your limits more often (while using CC) could make you want to upgrade plans, and a more efficient harness could keep you from doing that. But I can't imagine the cost (to Anthropic) of those inefficient tokens is worth it to them.

[epylar]

Subsidized Anthropic subscriptions seem to work fine on the oh-my-pi harness, somehow.

[3uler]

Calling per token usage of the US closed source labs has always been funny to me, we have a clear model of what it actually costs to host these models from open models.

Your subscription is not subsidised, it is just closer to the actual cost of the model…

[jeffrwells]

According to ccusage I use about $5k of tokens on my $100/mo Claude Max plan and only hit 5-hr windows where I have to switch tools for a couple hours about once a week

[epolanski]

It's so unfortunate people don't realize it's cheaper to write their own well working agent instead of insisting with general purpose bloated ones like Claude.

200$/month is a lot of money on Luna/DS4 flash, like really a lot and the results are much better than clowning on bloated CC.

It's absurd how you have more and more organizations encoding their processes on LLMs and "engineers" (charlatan coders) don't even bother optimizing the tool they use most.

[RussianCow]

> the results are much better than clowning on bloated CC.

I won't argue with the cost effectiveness, but the results are very much not better. Opus and Fable are in a different league than DS4 Flash. Even GPT Terra, which I really like overall, sometimes gets stuck in weird loops and starts to do stupid stuff once its context window fills up. Whereas I can more or less trust the big models to just Do The Thing™ on the first try.

With that said, you get way more value out of a GPT subscription than you do from Claude, partly because of the ability to use more efficient harnesses.

[trollbridge]

I have yet to spend $200 on DS4 after two months of using it with an entire team.

[bigstrat2003]

It's cheaper still, and just as effective, to write code yourself instead of having the LLM do it for you.

[visarga]

The system prompt part is surely cached across all users.

[yojo]

You still pay cache token costs on API calls. Cache cost/token are 90% lower, but you pay it every single turn.

I’m not sure if they let you skip the cache write cost on the first turn. That would imply cross-user caching infrastructure or special casing the default system prompt to give you a discount. Maybe? Away from the computer but you could try a “hello” in a fresh session and see what was billed.

[cyanydeez]

Its inexpensive to reserve KV cache for the first turn and would benefit users, given the first turn already requires costly locating and allocating a model slot for a user.

So yeah, when they banned tgird party harnesses there was a technical and $ case to have.

[dcrazy]

For Anthropic models, yes. But OP was using Claude Code with GPT.

[tosh]

the system prompt still takes up useful space in the context window and steers the model into unnecessary actions and over-thinking patterns

[tosh]

ty re --disallowed-tools

for coding agents 'shell' is often all you need (just make sure the environment has the necessary tools)

[colourclash]

appreciate the tip, kind netizen

[bonestamp2]

> Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.

Do we have any insight into whether it is actually spam and not useful info such as project or programming language specific context?

[tosh]

in my book anything that is (I'm sure well intentioned) and injected to help the agent — but doesn't help it — is a waste of tokens

but even injected context that when I read it sounds useful can oversteer the model and make it second guess or take a more complicated route than it normally would

(you can see this when looking at traces with and without that injected context)

often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help

that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary

'spam' is maybe a harsh way to say it

unfortunately I don't see an easy way other than to invest time and tokens into finding out which parts of the added context (in system prompt, injected in turns etc etc) are actually helpful or harmful and when

I'm just doing the easiest thing I could think of: start from nothing or close to nothing

that seems to work better than what most harnesses are doing

turns out GPT 5.6 Sol is all you need

[kelnos]

> in my book anything that is (I'm sure well intentioned) and injected to help the agent — but doesn't help it — is a waste of tokens

How do you tell, though? I guess what I'm asking is: the data you presented shows some crazy differences, but the token burn alone doesn't tell us enough. What was the output of the task like? Did the harnesses that burned fewer tokens give you as good a result as the ones that burned more?

I guess it's subjective, of course, but nearly everything about LLM use is...

[tosh]

I agree, only looking at the token burn is not enough

in this case it was 10 tasks and all harnesses could complete the tasks successfully, of course now the question is: will this hold for more and more complex tasks but I had to start somewhere :)

Checksum: Compute a file’s SHA-256 checksum and save the exact digest to an output file.

Log correlation: Correlate nested service logs to identify and summarize a request’s complete execution path.

CSV report: Parse quoted CSV data and aggregate paid orders and exact decimal totals by region.

JSONL join: Join related JSON Lines datasets and produce a correctly grouped and ordered report.

Archive repair: Find the correct version of a corrupted file in a tar archive and restore it.

SQLite migration: Safely migrate a SQLite database schema and verify the resulting data and constraints.

Python bug fix: Repair an interval-merging implementation so it passes visible and hidden edge-case tests.

Python CLI: Implement a robust command-line program that reads JSONL and reports validated statistics.

Multi-file feature: Add an atomic feature across a small Python package, CLI, and associated tests.

Pipeline repair: Fix a Make, shell, and Python reporting pipeline so it handles general input and passes verification.

[mikodin]

I am super curious how much in your opinion the extra prompting is useful?

>often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help >that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary

I purposefully do this as I imagine it is useful. In my project I am seeing solid adherence to norms and a deep capacity to iterate on completed features. Essentially for each feature I have the model make a {featureName}.AGENTS.md at the root folder of the feature, where it maintains what is going on.

I am moving between Claude Code and Codex atm, but I began this pattern when pi + kimi 2.6 was my main driver.

[tosh]

I do think prompting and reference files (e.g. for architecture, tech stack, …) can be extremely helpful. I also do this in my projects (challenge is keeping drift of these documents at bay).

What I wanted to emphasize is that whatever is in the context (whether system prompt or user message does 'steer' the model in a strong way, so everything in the context affects overall performance in a way. Even if it is 'just' net neutral it takes space up in the context window.

The context window is very very precious, everything that goes into it should help (not just hopefully help).

The challenge is coming up with good stuff to put into that context. A good agents.md file will be better context than whatever the popular harnesses have in their system prompt.

Also good to keep in mind that newer models are very good and more agentic than older models so they are better at exploring their environment based on the tasks you give them.

[cyanydeez]

I tailored an opencode agent with terse, minimum instruction, going as far as using ascii symbols and definitions.

Works fine. My conclusion is treat it like a starting point and not a holy bible of everything the model needs to know.

I pair this with making sure task files are commented with headers like build scripts so context+instruction is consistent.

The happy path then is the LLM pulling in context where it needs it.

[trollbridge]

To be fair, 5.6 Sol is pretty much doing a ton of reasoning you can't see (and which you don't think about the cost of if you're using it via an OpenAI Pro subscription)

[nijave]

On Claude Code you can run /context and it'll explain everything

[bermudi]

If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi?

The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.

[tosh]

smol only has 1 tool: sh

the system prompt of smol is shorter than the system prompt of Pi

smol has no system prompt

system prompt of Pi 0.83.0

""" You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files.

Available tools: - read: Read file contents - bash: Execute bash commands (ls, grep, find, etc.) - edit: Make precise file edits with exact text replacement, including multiple disjoint edits in one call - write: Create or overwrite files

In addition to the tools above, you may have access to other custom tools depending on the project.

Guidelines: - Use bash for file operations like ls, rg, find - Use read to examine files instead of cat or sed. - Inspect PI_* environment variables for current model and session details. - Use edit for precise changes (edits[].oldText must match exactly) - When changing multiple separate locations in one file, use one edit call with multiple entries in edits[] instead of multiple edit calls - Each edits[].oldText is matched against the original file, not after earlier edits are applied. Do not emit overlapping or nested edits. Merge nearby changes into one edit. - Keep edits[].oldText as small as possible while still being unique in the file. Do not pad with large unchanged regions. - Use write only for new files or complete rewrites. - Be concise in your responses - Show file paths clearly when working with files

Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI): - Main documentation: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/README.md - Additional docs: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/docs - Examples: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/examples (extensions, custom tools, SDK) - When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory - When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), pi packages (docs/packages.md), environment variables (docs/environment-variables.md) - When working on pi topics, read the docs and examples, and follow .md cross-references before implementing - Always read pi .md files completely and follow links to related docs (e.g., tui.md for TUI API details) Current working directory: /workspace """

[azuanrb]

What tools are you using to run the comparison? Or are you just running the prompts and checking them with something like ccusage?

I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t.

Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.

[tosh]

I built an ad-hoc custom comparison framework to inspect system prompts, caching behaviour, tool call outputs, exact api requests and responses and so on

I agree fewer tokens is not necessarily better but a bit counter-intuitively often the harness using fewer tokens is not only done faster but has better results

(that said: of course check the results, look at the full traces, agree!)

[azuanrb]

Thanks! I ended up built my own too. Just thought there are other better options out there that I might've missed.

[Groxx]

Running some local models and wondering what insanity was consuming 100k+ tokens to respond to "hello world" was rather eye-opening, yeah. Full of pointless fluff.

[cyanydeez]

I think prompt engineering went way to far into verbosity when at the same time agent and tool calling are much more important. Now a small system prompt can grab the contect it needs based on user context before bloating. On local models, i use dynamic context pruning.

[adoltech]

> The context window is very important.

It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.

[nijave]

There's an env var you can set to tell it when to compact. It's especially helpful for models with big context that lose coherence much earlier.

[tosh]

I will look into it more to see if I have configured it wrong but I think the token efficiency hurts cache use as caching only starts at 1024 tokens so for tasks where smol is under or close to 1024 tokens most of them are uncached

[chrisweekly]

Thanks for sharing. Naming tangent: FWIW I have recent interest in "smolvm" microvms from smolmachines.com, which are unrelated.

[brettgriffin]

Anything else I can read about smol?

also, are you using a tool to collect those metrics? what is it?

[tosh]

smol is basically this 9 line python agent re-implemented in Go

https://news.ycombinator.com/item?id=49006862

I'll have more about it in the next hours/days, you can follow me on twitter in the meantime (https://x.com/__tosh)

[growt]

So smol is your agent? Do you have a GitHub link?

[tosh]

coming in a few hours

you can follow this org in the meantime https://github.com/smol-env

or on twitter here: https://x.com/__tosh

[try-working]

that's why i recommend running everything through a router.

[skeptic_ai]

I think it’s too easy to trick you into token counting by checking tokens used. I think we should only focus on Price, time and productivity. Some kind of ratio

So a cheap model with many tokens or an expensive model with much less tokens could potentially be same price.

Maybe only time will be the differential factor.

[esafak]

Same task? How did the results compare?

[tosh]

the tasks were all simple agentic tasks

like creating a checksum of a file, merging csvs and so on, fixing a makefile pipeline

with known 'good' outcomes

all harnesses could reach the outcomes, only cost, time, number of tool uses and so on were different

(Claude Code failed once in 1 task but I think that was just an unfortunate outlier, the tasks aren't that difficult)

[applfanboysbgon]

While there's a lot of bloat in the harnesses, almost all of it is cached. Your harness appears to be caching almost nothing. That will almost certainly bode extremely poorly for longer-running tasks than the simple ones you mentioned testing. Benchmaxxing for simplistic tasks could leave you not only suffering in completion rate on complex tasks but also possibly even not end up with the cost gains you're hoping for. And maybe that's fine if the only thing you use agents for is the very simple tasks, which is certainly a defensible decision, but probably bears mentioning when you're comparing it against harnesses with a different design goal.

[tosh]

smol is also prefix caching

the uncached tokens are also from runs where smol finished a task below 1024 tokens (the minimum amount of tokens needed to activate caching) which is less tokens than other harnesses are using for their system prompt (!)

> GPT-5.6 and later models: Caching is available for prefixes containing at least 1,024 tokens. This is a strict minimum.

https://developers.openai.com/api/docs/guides/prompt-caching

so in this specific case the count of uncached tokens for smol makes it look worse than it actually is

that said: it does makes sense to add more tasks that are difficult enough to fill the context window to compare the harnesses for how well they deal with compaction

staying below compaction (or with compaction at fewer compactions) is not only cheaper and faster, it also helps the agent stay on track

[applfanboysbgon]

Yes, I do agree that avoiding compactions is extremely desirable, just make sure you're actually testing for how well that happens and the results!

[tosh]

will do!

[xienze]

> There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment

Be careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.

[tosh]

agree, that makes it a bit tricky to compare (esp if you also want to add different models and reasoning levels into the mix)

I will add more tasks (esp longer ones) and think more about grading, the current tasks were easy to grade because the desired outcomes are well specced but I will also look into more open ended tasks and how to grade those

thank you!

[zuzululu]

this benchmark is a very misleading because it doesn't necessarily point to these harnesses saving usage and much of the harness that comes with codex for example is designed at cross functional agentic usage which you will be leaving behind by using a different harness. openai spends a lot of time as does claude. also note that cached is what makes up bulk of the total and codex and claude has huge discounts in that scenario.

the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.

[tosh]

smol currently is very simple so it definitely does less things, like no subagent orchestration

I will look into how token usage looks like for longer sessions and more complex tasks

re caching: the cache ratio for this bench looks 'bad' for smol because it often finishes a task before caching kicks in (caching starts at 1024 tokens)

thank you for flagging this

[zuzululu]

sorry i didn't mean to call you out just wanted to point out at the comments thinking this means using sol/pi you can get many fold savings in tokens which isn't true

smol looks interesting i think it could have a potential niche although tokens are only going to get cheaper and cheaper here