[runako]

re: the comments that Opus is not cost effective...The whole sales pitch behind these tools, and quite specifically the pitch OpenAI made yesterday, is that they will replace people, specifically programmers. Opus is cheaper than a US-based engineer. It's totally reasonable to use it as the benchmark if it's best.

Also keep in mind that many employees are not paying out of pocket for LLM use at work. A $1,000 monthly bill for LLM usage is high for an individual but not so much for a company that employees engineers.

[michaelt]

My experience with coding agents is they need a lot of hand-holding.

They're impressive despite that. But if Sonnet is $20/month and I have to intervene every 3 minutes, while Opus is $100/month and I have to intervene every 5 minutes? ¯\_(ツ)_/¯

[epolanski]

> My experience with coding agents is they need a lot of hand-holding.

So do engineers.

The difference is that IRL engineers know a lot about the context of the business, features, product, ux, stakeholders, expectations, etc, etc which means that the hand-holding is a long running process.

LLMs need all of these things to be clearly written down and specified in one shot.

[runako]

Really depends on who's paying the bill, and how much gets done between interventions, right?

Inverting the problem, one might ask how best to spend (say) $5,000 monthly on coding agents. I don't know the answer to that.