[gozucito]

The meat of the report for SWEs:

SWE-Bench Pro Sol: 64.6% Fable: 80% Opus: 69.2% (!!!!)

So, it still trails Opus, significantly, and is not a next-gen coding model like Mythos/Fable 5.

Disappointing to say the least, but somewhat expected.

[osti]

SWE-Bench pro is pretty much useless now even though many ppl still look at it. OpenAI published a report yesterday saying so as well. Only look at DeepSWE and FrontierCode right now for coding imo.

[SirMaster]

Amazing, a company that does poorly in a benchmark says that benchmark is useless...

[osti]

SWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors.

But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.

[tripleee]

> Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark

yeah that was the point of introducing the Pro version

[sk4rekr0w]

You've overstated the conclusion. The SWE-bench series has had issues since its inception.

OpenAI no longer recommends SWE-Bench-Pro as a benchmark: https://openai.com/index/separating-signal-from-noise-coding...

[bigfishrunning]

That's smart, they only recommend benchmarks that make them look better then their competitors.

[paxys]

Makes sense why they released an entire study yesterday discrediting SWE-bench Pro.

[osti]

And they'd be right, it's an almost saturated benchmark where even some subpar open source models score very well on. And most models are clustered within a small range so it really doesn't tell you much.