[marginalia_nu]

It's my hunch that benchmarks are often in practice very difficult to get to actually measure the right thing, because of the inbuilt conflict in requirements of isolating the thing we want to benchmark to make sure we benchmark the right thing; and having the benchmark still resemble anything like a real-world scenario.

They also tend to sort of make you want to optimize the way the code already works on some level (a function or a service), as opposed to making larger scale optimizations that often yield much bigger performance improvements. Often the really big real world improvements are architectural rather than the result of tweaking the access order to get better CPU cache access patterns. Like the latter happens too, but you run out of those types of optimizations relatively quickly.

Like if you have some small piece of code you want to optimize (for some reason), a benchmark is invaluable; but if you have a system you want to optimize, it's less useful.

[davecb]

I entirely agree: I used to run the performance team at Sun Canada, and 99% of the time, we were on a bottleneck-hunt. Annoyingly, the bottleneck was rarely in the first place we looked (;-))

Benchmarks are well-known, well-understood and popular. They aren't what capacity planners and performance engineers use, though.