[fastball]

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

[ValentineC]

Medium or low supposedly prevents Opus 5 from overthinking:

https://xcancel.com/danshipper/status/2080700057892815114

[conception]

https://cognition.com/frontiercode

Quality vs cost - medium is the sweet (perhaps better too!) spot.

[fastball]

That is just a single benchmark tho

[dhorthy]

my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward

[sothatsit]

If I need something smarter I use Fable. Medium works well and is quick. Opus 5 medium feels much better to me than Opus 4.8 medium.

[dhorthy]

yeah someone will have to re-run this bench on various effort levels. unfortunately it is not cheap