[onlyrealcuzzo]

None of these tools measure how effective they are...

It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to...

Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much.

Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or some major player would bundle tools like this into the CLI / harness.

AFAIK, none of the major players do. That's a sign to me these don't work in general.

I've tried building some tools specific to bug fixing. Intelligently feeding context massively helps smaller models. But, what I've found - surprisingly - is that a smaller, much better focused, including a lot of helpful data as well, has almost no impact on larger models compared to what they do by default.

You do save some tokens, though, which is what they're claiming - but not ~99%...

[hansvm]

> otherwise, popular solutions would integrate the idea

None of the major players are incentivized to care about this, especially not over other opportunities. Why would you expect them to integrate it?

One of the biggest wins you can institute for your own codebase if you use agents is writing your own harness, by a huge margin. The defaults are fine, but you can do better.

[Cpoll]

They're incentivised because they're offering plans at a loss and/or pricing out potential customers. All these LLM companies are competing on accuracy and price.

[onlyrealcuzzo]

> The defaults are fine, but you can do better.

Why can I do better than Pi?

I don't want to build my own harness and deal with the bugs... I want to build my project...

My understanding is that Codex / Claude / Gemini subscriptions don't work with custom harnesses.

It's pretty hard to beat 5x more usage if you have the $200/mo subscription by using the API instead.

[smallerize]

If you're looking for an efficiency-focused harness, I had a pretty good time using the Dirac agent. The line-based anchors were slightly buggy though (this was a couple months ago) and would sometimes add the same line of code multiple times or leave an anchor in the output.

[GodelNumbering]

Creator of Dirac here. Came across this too late. The behavior you mentioned happens more commonly in smaller models, rarely in larger/frontier models. The underlying code is clean but the smaller models often make boundary errors (off by one type). Which models did you observe this with?

[smallerize]

That tracks, this was a quantized version of GLM 4.7 Flash. I was surprised because I though this specific thing was not supposed to happen lol.

[hboon]

Codex definitely does and Claude Max definitely doesn’t.

[Bnjoroge]

definitely doesnt is a strong word. it technically is possible, but you might get banned

[hboon]

That was true. But actually, I think that's changed a few weeks ago since they introduced a API credit amount equivalent to your (eg. $100, $200) that will be used for such cases. So they don't ban you, they just bill you that allocated credit and then actual API cost.

[Bnjoroge]

Yes. That’s possible in addition to using your actual subscription. I’ve been using it via cliproxy for all harnesses and even my own code review agent hooked up to github apps. Not banned yet but I also dont do crazy stuff with openclaw or hermes

[doix]

It's too hard to define what "works" even means in this case. Look at the example savings output. A lot of it is kubectl output.

Your suggestion to using coding benchmarks doesn't really capture the whole picture. I haven't seen a benchmark using kubectl.

> AFAIK, none of the major players do. That's a sign to me these don't work in general.

It's a lose/lose for major players. If it works well, it will lower their revenue. Also there's a high risk it'll significantly worsen results for some people, even if it improves results for others.

[taude]

I don't think frontier model providers are going to be incentivized to invest in this much, yet. Once inference gets more competitive, sure. I haven't looked lately, but won't be surprised if tools like OpenCode do do what you're suggesting, though. Third-party coding harnesses ARE aligned to deliver this type of feature and optimization.

[no-name-here]

> I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or some major player would bundle tools like this into the CLI / harness.

VS Code launched it as a feature in their bundled AI functionality last month: https://code.visualstudio.com/updates/v1_121

[onlyrealcuzzo]

Bundling implies interest...

Defaults imply working...

[irthomasthomas]

My partial solution to this was to store the full response in a file and prompt the agent to read that if the condensed version had stuff missing.

[unphased]

So often we will burn 20% of limit in a single ill conceived agent tool call that we're simply not going to be able to or want to be able to intercept. Where I see a tool like this being a real step forward is to add a decision point. it does not have to bubble up to hard-require user to provide permission, but it can let the LLM have an intermediate checkpoint to say that it's about to get blasted with 30k tokens and here is roughly the shape of it and do you wanna adjust or whittle it down if you know what you're looking for etc.?

There is definitely tons of value to extract from this line of thinking.

[varispeed]

You can't measure effectiveness, because you never know what kind of model will process your prompt. One request you might get full e.g. Opus and another they'll downgrade it to Sonnet or something more basic. I have this with "Opus 4.8" all the time.

[jahala]

This is the reason, when I built a tool in the same space, I chose to benchmark with cost per correct answer.

Reducing tokens and also turns is quite worthless if the LLM doesn’t solve what you put it to do.

[esafak]

Did you benchmark the competition and can we see?

[jahala]

No I don't have the funds to benchmark the competition, but would be happy to put the numbers up if any token whales feel like having a go.

https://github.com/jahala/tilth/tree/main/benchmark

[alex7o]

Oh that is a nice approach whish more benchmarks did cost per successful

[onlyrealcuzzo]

The problem even attempting to develop a tool for the frontier model space is that the cost to run a statistically significant benchmark is almost certainly going to be over $100 - for a single model.

Unless something is like 25%+ more cost effective on Gemini for a task, I would not assume those savings are going to transfer to GPT.

If you need to run a test this expensive and slow for every release, hobbiests aren't going to do it.

And if you wanted any broadly specific improvements to coding like they all claim, the costs would be in the thousands per release even for a single for a single model.

And they almost certainly would not be eye popping.

If the models could be SUBSTANTIALLY better, Google and Anthropic and OpenAI wouldn't be finding that out from a hobbiest making wildly unscientific claims.

[jahala]

Yup, this is hitting it on the nose. But, despite the cost - the benchmark is the vital ingredient that cant be skipped. Otherwise, you don't know if what you're building is actually helping the agent rather than hindering it.

On the previous large benchmark run, i proved 40-50% cost reduction per correct answer.

I'm not sure why the vendors aren't using token filtering/compression more in their tooling, but perhaps they don't mind users feeding them more data and using more data.