[platinumrad]

Maybe I'm a control freak, but asking agents to one-shot random apps is nothing like how I actually use AI in software engineering.

[ValentineC]

One-shot benchmarks are great for me as a solo creator, since they slightly correlate to whether the better frontier models (Opus and Fable for me) make better decisions about things I didn't spec, or whether they'll give me better suggestions right off the bat.

[anuramat]

just found a decent looking benchmark for iterative development: https://swe-milestone.com/

surprised it isn't a bigger thing, eg artificial analysis doesn't report anything like that

still doesn't measure the human-agent interaction part, but that's pure vibes atp

[fragmede]

It's not, but it's trying to bring any level of objective measure in this realm, vs just going off of vibes.

[mikeocool]

Yeah, the models have all been really good at generating greenfield apps for a really long time (in the scope of LLM time).

I suppose it’s interesting to see how they make better greenfield apps. But I am much more interested in how they solve hard problems in existing gnarly codebases.

[anuramat]

I imagine one could one-shot a basic app and then feed feature requests one by one, sounds like an obvious way to benchmark architecture/maintainability

[bhu8]

Absolutely yes, but that's how you become twitter/X famous

[thsbrown]

Man I do ponder this all the time.