[fastball]

On the one hand: yes, pelicans on bikes are definitely in the training set at this point.

On the other hand: the test is clearly not saturated, given that you can see a clear difference in output at the various reasoning levels / model versions.

[pupppet]

I sort of agree, but within the same model I expect the reasoning effort to be reflected in the quality of output and that's basically how it played out. When you're comparing different models, then it's just who benchmaxxed the best and there's not a lot of value there.

[fastball]

But that is my point: if benchmaxxing was all the labs were doing, then surely the dumber model could/would have equivalent performance? Rather than noticeably worse perf on a (somewhat trivial to game) test.