[jupp0r]

Using CPU utilization as a performance metric can be extremely misleading. My favorite article on the subject is from Brendan Gregg:

http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-...

A much better way to test the influence of the new compiler would be to test the actual throughput at which saturation is achieved (which is what the benchmark in the C++ grpc library measure to assess their performance).

[dkhenry]

There is a fairly robust set of benchmarks that are run to test out performance improvements[1] and macro benchmarks are the ultimate test of holistic improvement. CPU isn't a great proxy, but one of the biggest problems in real world performance on this specific system ( databases in general ) is latency. CPU time is a really good proxy for latency so by taking a look at CPU time we can get an idea of how the system will respond under "normal" conditions.

1.https://benchmark.vitess.io/macrobench

[et1337]

In this case the regression also caused a 3% decrease in throughput.