[meteorfox]

This part I'm skeptical:

  "Compared to CPU time, CPU instruction is a better metric,
  as it reports the same numbers regardless of CPU models
  and CPU loads for the same request."

CPU instructions will be more stable than CPU time for sure, and it does show that their metric is stable based on their chart but a single CPU instruction can take multiple cycles, especially if there are stalls in the pipeline or other processes are "polluting" the cache. Depending on the CPU model, the number of uops that can be issued concurrently varies, and the latency of these instructions will also vary on the size, and access pattern to their memory hierarchy.

Also, what about a change in compiler version? That can also vary the number of instructions. Unless they are referring to a Python bytecode instruction as a CPU instruction.

Would measuring CPI be a better indicative of their efficiency? They could also track both, no need to settle for one.

[zerohp]

I agree with your skepticism. They will not detect a performance regression when there is a change in instruction mix or cache miss ratio.