[levocardia]

Depends a lot on the task demands. "Got 95% of the way to designing a successful drug" and "Got 100% of the way" is a huge difference in terms of value, and that small bump in intelligence would justify a few orders of magnitude more in cost.

[9dev]

But that objective measure is exactly what we’re lacking in programming: There is often many ways to skin a cat, but the model only takes one. Without knowing about those it didn’t take, how do you judge the quality of a new model?

[altern8]

I would say following instructions.

If Claude understood what you mean better without you having to over explain it would be an improvement

[snoman]

I agree with you, but my gut tells me that a lot of people don’t know what a good outcome should/could look like and are accepting whatever it delivers.