[Retric]

Companies game benchmarks, but sure they are better than nothing.

[MattRix]

I mean you wanted something objective, and they are. I don’t know why you’re being dismissive of them, they’re a huge element of what drives model development forward.

These companies aren’t just making stuff up, they really do want to improve the models, and the models really are improving.

[Retric]

> I don’t know why you’re being dismissive of them

I’m aware of multiple cases of benchmark cheating/“optimization”. So, taking benchmarks a face value seems laughable.