[bsaul]

bradfitz : "That was one of the biggest architectural changes in the net/http.Server in quite some time. I never did any benchmarking (or optimizations) after that change. "

Sorry, what ? It's not like the http server of the stdlib is here only for doing hello world code samples... You would imagine those benchmark to be part of some CI process along with unit tests.

[lmb]

Benchmarks in CI are hard, because you need them to run in the exact same environment to make any sort of conclusions. But CI's are often noisy, virtualised, dockerized, whateverized. There is not much benefit in that.

[poooogles]

I encountered this a while ago, I've ended up bechmarking in relation to a past change and ensuring the percentage difference it's within a margin.

Not ideal but it's better than pure X vs Y.

[bsaul]

That could be an idea for a service. Provide an instance type with stable execution environment for benchmarking. Just stable, and not necessarily performant.

[bluejekyll]

I think it's harder than that though. You're trying to predict real-world numbers, and a hobbled test environment could show benchmark hotspots which might never occur on a real system.

It could still be a good, but very expensive, service, but its hard for me to imagine something that would be one-size-fits all service that would accurately predict real world experience.

[bsaul]

At least it could catch some easy regression between version after some heavy refactoring, like this particular one. Better than the complete random performance of current VMs on cloud.

It wouldn't be much harder than just making sure you're the only VM running on the hardware (disk & cpu) . Much like a "reserved instance", with real guarantees against side effects.

[bluejekyll]

I think it could catch this, but it could just as easily miss that there was a huge degridation on HDD vs SSD; high ram vs. low ram; fast vs slow CPU; GPU vs no GPU.

Obviously you can build test suites for each of these scenarios, but I think it would be expensive to run all of them. That's all I'm really saying, it's by no means a bad idea, I think it's a great idea, just going to require some upfront thought of what type of environment the software is going to run in.

[icebraining]

Why would it be very expensive?

[bluejekyll]

I'm assuming that you'd run many benchmarks, not just "Hello world". Each one will run for some number of cycles, to make sure that you have a good mean from each run.

So it's expensive b/c it consumes CPU and time on shared systems, and those cost money... so I think a service like this could potentially cost significantly more to operate than say travis-ci.

[jobvandervoort]

Talking about this commit: https://github.com/golang/go/commit/faf882d1d427e8c8a9a1be00...