[dan-robertson]

The commonly used benchmarks are often pretty useless or irrelevant. Coming up with useful benchmarks is very hard and a pretty tragic thing about the supposedly irrelevant independent CPU benchmarking of the past 20 years is that CPUs are specifically designed to perform well at the silly benchmarks that are popular as well as at real world workloads where I think CPU manufacturers will have their own suites of more carefully designed more realistic more relevant benchmarks.

My claim is that (1) results on irrelevant benchmarks like SPEC do matter for CPU sales (probably not for big tech companies that operate their own datacentres – they’ll likely do their own testing – but likely for many ‘savvy’ consumers and also for companies that want to market their computers as having fast CPUs) and (2) the complicated world of benchmarks is handled very poorly by the people who tend to evaluate CPUs with them and publish their results.