[beering]

Really awful how the AI labs are skillmaxxing /s

Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.

[dbt00]

It's a problem because of Goodhart's law.

If you train towards the test, you aren't necessarily improving overall fitness, but you are destroying the value of that test over time because you're decreasing its correlation with overall fitness.

[Dylan16807]

Avoiding "benchmaxxing" helps keep benchmarks be a good quantitative way to compare models! It's hard to come up with new benchmarks, and it's a waste of everyone's time if we have to churn through them constantly. Effort to keep AI labs working on general skills and not "teaching to the test" is effort well spent.

[Wowfunhappy]

> If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark!

I don't think I'd go that far!

When someone says a model has been benchmaxxed, what they really mean is that it performs better in benchmarks compared to their real world experience. That's a real thing, I've certainly experienced it with some models.

...my take is that some things in life just resist quantitative measurements. Who is the best job candidate? What is the best programming language? Add AI models to the pile.