[V1]

Because we people obviously don't have a datacenter in our own basement. And the common mistake people make when benchmarking is running the servers on their own machine and then use the same machine to benchmark the server it's running.

You need to have multiple (powerful) machines for this. And also, spinning up machines in the cloud is quite easy to do and allows people to reproduce the same test results because you have access to exactly the same environment.

[snaky]

>You need to have multiple (powerful) machines for this

Oh really? For simple http 'hello world' comparision (where you are interested in relative numbers, not absolute ones) bechmark?

All you need is one old and slow laptop (with test contenders) and one modern and mighty (with test script). The only thing you have to be sure about is the test script can generate more load than test contenders can handle. Even if the old laptop isn't slow enough you can just add some predictable and stable load to cpu/disks/network/whatever is a bottleneck for them - you may use tools that are available for that or even quick & dirty hacks like one-liner 'while(1) {do some math}' that effectively make your 2-cores CPU 1-core while running with high system priority.

[devicenull]

You can reproduce them? So you can match the loads running on all the other VMs on the same host then?

[kalmar]

You could use cluster compute instances which use hardware virtualisation, and I think gives you the whole machine. With spot pricing, you could run them for about 21¢ an hour.