[AquinasCoder]

While I appreciate, they publish this information, it's increasingly hard to keep track of it all. I've lost the mental model of how different models at different effort levels perform and what tasks they are good at.

In practice, I tend to just use the default on Claude Code that works well enough. But I wonder to what degree other users really play around with these settings to optimize for their project.

[matheusmoreira]

I always use Opus 4.8 at max effort for everything. The $20 subscription didn't have enough tokens, but the $100 one had too many of them. So now I just max out Opus in order to maintain 100% weekly utilization.

[Abishek_Muthian]

I'm a senior skilled developer and I find Anthropic $20 + Open AI $20 + OpenCode Go $10 offers more value than $100 on any particular service.

Juggling between all different models/agents is quite simple with Zed.

A caution about OpenCode Go though, the entire company seems to be run by AI so there's lot of billing related issues with zero support. I subscribe new every month as I lost money due to double payment with automatic subscription.

For non coding related tasks I use local models.

P.S. If anyone is interested to read more about my setup, let me know I'll publish a blog post.

[theshrike79]

I've been running about the same stack for well over a year now, Anthropic cheapest + OpenAI cheapest + z.ai coder (black friday offer).

The Z.AI is a bit wonky, so now I'm moving to Openrouter for Qwen+Kimi+Deepseek?GLM

My summer project is to figure out a proper agentic system where a "big" model does the planning, but automatically uses a cheaper one for the grunt work. Having Opus to config edits is just stupid :)

[jurgenburgen]

> My summer project is to figure out a proper agentic system where a "big" model does the planning, but automatically uses a cheaper one for the grunt work. Having Opus to config edits is just stupid :)

My company pays for the tokens so I don’t care. Biggest model and max everything. The slight risk of a smaller model making a mistake is more expensive than just running the bigger model all the time.

[theshrike79]

Oh I do that at work too, for sure. I'm riding that Claude Teams subscription like I stole it.

But I'm playing the long game you see. The tokens will get expensive and the monthly subscriptions will either go away or also get too expensive.

Then companies want efficient token use and cost control - and I'll already know how to do that =)

[rdudek]

Would love to read that blog post. I'm toying with running local AI model with Claude and GLM as well depending on a task. Pretty decent success but it could be better.

[Abishek_Muthian]

Here's the blog - https://news.ycombinator.com/item?id=48748443 .

[kamaal]

>>For non coding related tasks I use local models.

What sort of hardware are you using to run local models? And how do you use them?

[Abishek_Muthian]

4090 RTX laptop and a M4 Mac Mini, I've posted more information in my blog mentioned in another comment.

[ido]

I’m interested!

[Abishek_Muthian]

I've posted in another comment above.

[easygenes]

I'm a heavy enough user that I have both the OAI and Anth $200 plans. I always use at least 50% of my weekly Opus quota at Extra setting (meaning I use double the limit of the $100 plan, at minimum). Max I rarely touch because it is twice as slow and the incremental capability gain is minimal. Usually if Opus can't sort something well at Extra, the answer isn't to use Max but to hand the issue off to GPT-5.5 at XHigh.

[tyg13]

I too have settled into a kind of dual Claude/GPT model setup. I will often use one to review the other's work, or critique the other's plan in some way. Sometimes I'll have Claude implement a feature one way, then have GPT do it the other way, then have them both review each other's implementation. Then synthesize a final plan from the previous implementations+reviews.

I might just be having fun with models, but I have actually noticed their capabilities vary somewhat, and so my (perhaps vain) hope is that by using both, one can catch each the other's blindspots. It's still unclear to me if that's consistently happening, but I am making substantial progress in my personal and professional projects, so something seems to be working.

[versteegen]

> Sometimes I'll have Claude implement a feature one way, then have GPT do it the other way, then have them both review each other's implementation. Then synthesize a final plan from the previous implementations+reviews.

I've done variants of this a number of times, but feel like it was a generally waste of my time to then have to compare them and write up which parts I liked or disliked: if the output is something substantial, each will have its pros and cons. Clear-cut wins aren't very common. Of course it could work well if we automated the whole thing with an orchestrator; you just need a model with actual good taste (according to your own preferences) ... so we'll have to compare all the models to find that one

[dirtbag__dad]

Yes, same, between the two of them I feel like results are just better because they have different priorities.

At the same time, I’ve invested in tooling that prints and lints architecture I want, so which model is less of an interesting decision, because the results tend to be very close.

[ATMLOTTOBEER]

Agreed I think your strategy is optimal. This is what I landed on as well

[vcf]

Me too, I rarely hit limits anymore on the $100 Max, except for the brief period with Fable

[chewz]

This is actually very counterproductive with Opus 4.8 - you are wasting a lot of time.

For Opus 4.8 training with overblown internal dialogue and second opinions - Max effort burns just tokens and wastes time without much value. Spinning wheels.

[matheusmoreira]

Even with Opus 4.8 I have to try pretty hard to maintain 100% weekly usage. I'm actively trying not to be efficient at this point.

Now that the ban is lifted, max effort Fable 5 is gonna solve this problem quite neatly. Fable to plan and review, Sonnet for the implementation.

Wait, never mind that. Subscribers will only have Fable for a week.

[jon-wood]

Why are you optimising for token use rather than getting whatever tasks you're trying to do done? Max effort Opus will happily sit there talking to itself for five minutes fixating on every tiny detail while Sonnet or even Haiku will just blast through the problem and be ready for the next one in seconds.

[matheusmoreira]

Because I'm paying $100/month for this, so I absolutely want to use all of those dollars. Any leftover weekly usage is a gift to Anthropic.

I am getting things done. I've made major progress on my projects, and even started new ones. My most requested tasks are: code review, brainstorming and research. The fixation on tiny details is exactly what I'm paying for.

[m-dot-reviews]

I've been plugging this perhaps too many times now, but I am trying to bootstrap a user-sourced corpus of exactly "what model is good at task X". So, not benchmarks, but high-level tasks. There's a bit of a ordering problem in that nobody wants to bother commenting on a site that has few comments - so PTAL and contribute if you can. https://model.reviews

[sanderjd]

What I want is a harness that knows how to optimize this kind of thing for me.

[nl]

In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it.

They are often used for reading code though.

To expand on this, while the "big model to write a plan, small model to write the specific code" idea is quite common it trips up on edge cases.

In theory the flow works like this:

- small fast models read lots of code, and pass details to the large model to write a plan

- large model takes those details and writes a detailed plan

- medium models write the code

The issue happens when the medium model hits something that the plan didn't take into account (which happens a lot - the big model didn't actually read the code). Then it has to either guess, or pass back to the large model.

If it guesses, the plan usually starts to fall to bits.

If it passes back to the large model, inevitable the large model has to start reading lots of code. In that case you are paying the expensive tokens to read so you might as well have it write the code too (many less tokens are written than are read)

It might be possible to get this to work, but I haven't seen anyone who has tried agentic work with frontier models be satisfied with this hybrid setup.

I'd note that Amp (mentioned above) is probably the leader in using multiple providers in a coding agent but still uses frontier models to write code.

[whatshisface]

> In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it.

That's not something I understand very well. The less expensive models will quite happily chug away at tasks, if the codebase is well-structured (small files help a lot) and your instructions are clear. In contrast, I've never seen a large model turn bad instructions (instructions that would cause a human to think before starting) into a result I liked. You can run small models almost 10-100x as long for the same price in dollars, which covers a lot of correction and adjustment.

Why does everyone say the trade-offs are rarely worth it?

[nl]

> In contrast, I've never seen a large model turn bad instructions (instructions that would cause a human to think before starting) into a result I liked

I think the distinction is here.

I expect my agent to build from product level descriptions. This might include specific special cases that I call out, but will rarely highlight existing special cases or edge cases - they already exist in the code, and I'd expect a programmer to make sure that behavior continues to work.

If a feature hits lots of these edge cases, the weaker model that is reading the code (aka Haiku) won't understand their significance, and will report back to the planning model incomplete or incorrect information.

The planning model (Opus - which hasn't actually seen the code remember!) will build a plan that is incorrect or incomplete and delegate coding to the mid level model (Sonnet) which will do it's best to make things work, without understanding the overall picture.

This is how you end up with slop - for example Sonnet reimplements things that already exist because it found one of the edge cases, but Opus had never known about it because Haiku didn't understand it.

It's possible that the new "agent teams" feature in Claude code can help with this. That keeps each agent alive with its context so they can ask each other things, but I haven't tried that enough to be sure - let alone with the specific model mix like this.

In your case, you are giving the Sonnet model specific instructions for what to implement mindlessly. I'd expect that to work well!

But that's not the same as the agentic workflow many other are using.

[sanderjd]

Great info, thanks!

[cunningfatalist]

You might want to check out Amp: https://ampcode.com/

[sanderjd]

I appreciate the suggestion! But it isn't clear to me, from reading their marketing site, what they bring to the table from this perspective. Can you give me a more targeted pitch?

[usef-]

This page buried in their docs is a bit better than the homepage imo: https://ampcode.com/manual#why-amp

I haven't used them in a while so my info may be out of date, but they tended to track whatever models were the best and auto-use them for each task (eg, one for planning, subagent for a code search, other frontier for implementing). Their CLI seemed very well thought out to make you do things "the correct way" -- for instance, `/handoff` instead of `/clear`.

[sanderjd]

Thanks!

[cunningfatalist]

Sorry for the late answer and the missing context. usef- is right, the manual is probably the better page to share. Amp tries to give you a plug and play experience, where you can always see the actual costs and models/effort are autoselected for you. Some of my colleagues are big fans and use it a lot. I also like it, but prefer OpenCode.

[sanderjd]

Great. Amp wasn't even on my radar! Appreciate the pointer!

[manojlds]

Which is your own harness and your own evals for your tasks I guess

[munk-a]

I don't demand a customized compiler for my code even if such a compiler could outperform gcc. There is a lot of value in focusing on correctness to an extreme degree even if the outcome might be suboptimal to something more tailored - a tool with a large customer base can justify more resources going into its maintenance.

[sanderjd]

Maybe. But that sounds like a large amount of bespoke work for what seems like a common problem?

[manojlds]

I was talking about enterprise agents and then realized the question is more about coding agents.

[sanderjd]

Ah I see! Yes, I was talking about a coding harness, not an enterprise agent. I entirely agree with you that your suggestion of driving it via evals is the right thing for that use case!

[nolok]

Same boat as you, and my answer is "... Except when I ask and overall or checkup task that is specifically heavy or overseeing in which case I use the maximum level" which lately meant ultracode.

I'm not going to play around with thinking level every request because the goal is to make me save time not spend it in a different setting menu.

[brobdingnagians]

I tend to run it on High and then step it up for problems where I'm noticing it struggles, bump it back down after. Sometimes I accidentally leave a session in Ultracode for a day and wonder why things are taking so long, but generally happy with the results.

[jbvlkt]

Exactly this is my problem with all AI tools. I want someone else to create working tools for me so I can focus on my product. It is the same with other tools. I do not want to spent huge amounts of energy and time to setup my IDE, operating system or desk layout. I guess it is too early to have that now.

[jerojero]

I think that's the whole selling point of lovable?

[jimbo808]

It's really not that much. It's a bit hard to make sense of it not because it's hard to keep track of, but because they are being deceptive and opaque about what you're actually buying, and the thing you're paying for is different from one day to the next, as they fuck around with the parameters to boost subjective performance during a launch, then quietly degrade the service to cut costs.

[j45]

Just because it’s hard to keep track of doesn’t mean it’s not relevant.

Playing around with learning the differences is incredibly helpful to schedule on ones calendar weekly for an hour or two, while saving links throughout the week to try out.

[tash_2s]

I also ended up using max effort/reasoning for both coding and general chat. They don't spend too much extra time on simple tasks these days.

[deadbabe]

There are token optimization consultants that can help organizations find the right balance of models for their employees to minimize costs.

[throwaway219450]

Same advice as ever? We call it context engineering now, but prompt engineering still matters a lot. Most of the failures I run into are unspecified assumptions made by the model that derails the conversation, but usually updating the first prompt fixes it. Opus in my experience is a bit better about checking assumptions, while Sonnet will plow on ahead. An example is mentioning a file that doesn't exist: Sonnet will go ahead and try to grep your entire hard drive for it. Opus will say it's not local and request the path.

I trust neither for general knowledge and I still find Opus giving me answers that are completely BS. But the token spend for Q&A is nothing compared to coding, so I always use Opus + a lot of thinking. For coding, I find Opus to be better value/token but I haven't done any sort of rigorous test.

[paulddraper]

It's almost like you want an automatically intelligent choice of your artificial intelligence.

Understandable frankly.

[jacooper]

Just use deepswe as a reference point.