That could be an idea for a service. Provide an instance type with stable execution environment for benchmarking. Just stable, and not necessarily performant.
I think it's harder than that though. You're trying to predict real-world numbers, and a hobbled test environment could show benchmark hotspots which might never occur on a real system.
It could still be a good, but very expensive, service, but its hard for me to imagine something that would be one-size-fits all service that would accurately predict real world experience.
At least it could catch some easy regression between version after some heavy refactoring, like this particular one. Better than the complete random performance of current VMs on cloud.
It wouldn't be much harder than just making sure you're the only VM running on the hardware (disk & cpu) . Much like a "reserved instance", with real guarantees against side effects.
I think it could catch this, but it could just as easily miss that there was a huge degridation on HDD vs SSD; high ram vs. low ram; fast vs slow CPU; GPU vs no GPU.
Obviously you can build test suites for each of these scenarios, but I think it would be expensive to run all of them. That's all I'm really saying, it's by no means a bad idea, I think it's a great idea, just going to require some upfront thought of what type of environment the software is going to run in.
I'm assuming that you'd run many benchmarks, not just "Hello world". Each one will run for some number of cycles, to make sure that you have a good mean from each run.
So it's expensive b/c it consumes CPU and time on shared systems, and those cost money... so I think a service like this could potentially cost significantly more to operate than say travis-ci.
It could still be a good, but very expensive, service, but its hard for me to imagine something that would be one-size-fits all service that would accurately predict real world experience.