[falaki]

Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench.

[fmind-dev]

It takes time and effort to build such benchmark. It works at Databricks scale, I'm not sure smaller companies are ready to invest on internal benchmarks.

But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view.