[Workaccount2]

It's clear though that as the models get bigger and more advanced, their "reasoning" benchmark results improve. The conclusion though just focuses on the bottom tier models. The fact they even set out to create an LLM benchmark and only focus on bottom tier models itself is ridiculous.

The authors did the equivalent of "Lets design a human intelligence benchmark, and use a bunch of 12 year olds as reference points"

I will eat my hat if the authors rescind the paper in a year or so if their benchmarks show no difference on SOTA models.