[fibonacci112358]

Maybe OP was thinking about compilers "cracking" certain SPEC benchmarks: implementing exactly the optimization needed to boost a benchmark quite a lot, but that opt. probably won't apply to any other code out there (usually it's so targeted and risky with general C/C++ code that intentionally it doesn't work on anything else). That happened a couple of times over the years, I know about the Intel compiler cases for ex. I can certainly see LLM providers adding tricks that help a certain class of benchmarks, but doesn't help much for anything else.

[wtallis]

Intel's done it again recently, this time targeting Geekbench: https://www.intel.com/content/www/us/en/support/articles/000...

Both that and the SPEC compiler shenanigans are cheating by changing the test, not just over-specializing the product being benchmarked.