[mosselman]

Hyperfine is great. I use it sometimes for some quick web page benchmarks:

https://abuisman.com/posts/developer-tools/quick-page-benchm...

As mentioned here in the thread, when you want to go into the single ms optimisations it is not the best approach since there is a lot of overhead especially the way I demonstrate here, but it works very well for some sanity checks.

[Sesse__]

> Hyperfine is great.

Is it, though?

What I would expect a system like this to have, at a minimum:

  * Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions)
  * Multiple stopping points depending on said statistics.
  * Automatic isolation to the greatest extent possible (given appropriate permissions)
  * Interleaved execution, in case something external changes mid-way.

I don't see any of this in hyperfine. It just… runs things N times and then does a naïve average/min/max? At that rate, one could just as well use a shell script and eyeball the results.

[sharkdp]

> Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions)

This is not included in the core of hyperfine, but we do have scripts to compute "advanced" statistics, and to perform t-tests here: https://github.com/sharkdp/hyperfine/tree/master/scripts

Please feel free to comment here if you think it should be included in hyperfine itself: https://github.com/sharkdp/hyperfine/issues/523

> Automatic isolation to the greatest extent possible (given appropriate permissions)

This sounds interesting. Please feel free to open a ticket if you have any ideas.

> Interleaved execution, in case something external changes mid-way.

Please see the discussion here: https://github.com/sharkdp/hyperfine/issues/21

> It just… runs things N times and then does a naïve average/min/max?

While there is nothing wrong with computing average/min/max, this is not all hyperfine does. We also compute modified Z-scores to detect outliers. We use that to issue warnings, if we think the mean value is influenced by them. We also warn if the first run of a command took significantly longer than the rest of the runs and suggest counter-measures.

Depending on the benchmark I do, I tend to look at either the `min` or the `mean`. If I need something more fine-grained, I export the results and use the scripts referenced above.

> At that rate, one could just as well use a shell script and eyeball the results.

Statistical analysis (which you can consider to be basic) is just one reason why I wrote hyperfine. The other reason is that I wanted to make benchmarking easy to use. I use warmup runs, preparation commands and parametrized benchmarks all the time. I also frequently use the Markdown export or the JSON export to generate graphs or histograms. This is my personal experience. If you are not interested in all of these features, you can obviously "just as well use a shell script".

[Sesse__]

> This is not included in the core of hyperfine, but we do have scripts to compute "advanced" statistics, and to perform t-tests here: https://github.com/sharkdp/hyperfine/tree/master/scripts

t-tests run afoul of the “no Gaussian assumptions”, though. Distributions arising from benchmarking frequently has various forms of skew which messes up t-tests and gives artificially narrow confidence intervals.

(I'll gladly give you credit for your outlier detection, though!)

>> Automatic isolation to the greatest extent possible (given appropriate permissions) > This sounds interesting. Please feel free to open a ticket if you have any ideas.

Off the top of my head, some option that would:

* Bind to isolated CPUs, if booted with it (isolcpus=) * Binding to a consistent set of cores/hyperthreads (the scheduler frequently sabotages benchmarking, especially if your cores are have very different maximum frequency) * Warns if thermal throttling is detected during the run * Warns if an inappropriate CPU governor is enabled * Locks the program into RAM (probably hard to do without some sort of help from the program) * Enables realtime priority if available (e.g., if isolcpus= is not enabled, or you're not on Linux)

Of course, sometimes you would _want_ to benchmark some of these effects, and that's fine. But most people probably won't, and won't know that they exist. I may easily have forgotten some.

On the flip side (making things more random as opposed to less), something that randomizes the initial stack pointer would be nice, as I've sometimes seen this go really, really wrong (renaming a binary from foo to foo_new made it run >1% slower!).

[sharkdp]

> On the flip side (making things more random as opposed to less), something that randomizes the initial stack pointer would be nice, as I've sometimes seen this go really, really wrong (renaming a binary from foo to foo_new made it run >1% slower!).

This is something we do already. We set a `HYPERFINE_RANDOMIZED_ENVIRONMENT_OFFSET` environment variable with a random-length value: https://github.com/sharkdp/hyperfine/blob/87d77c861f1b6c761a...

[renewiltord]

Personally, I'm all about the UNIX philosophy of doing one thing and doing it well. All I want is the process to be invoked k times to do a thing with warmup etc. etc. If I want additional stats, it's easy to calculate. I just `--export-json` and then once it's in a dataframe I can do what I want with it.

[bee_rider]

What do you suggest? Those sound like great features.

[Sesse__]

I've only seen such things in internal tools so far, unfortunately, so if you see anything in public, please tell me :-) I'm just confused why everything thinks hyperfine is so awesome, when it does not meet what I'd consider a fairly low bar for benchmarking tools? (“Best publicly available” != “great”, in my book.)

[sharkdp]

> “Best publicly available” != “great”

Of course. But it is free and open source. And everyone is invited to make it better.

[llimllib]

I find k6 a lot nicer for HTTP benching, and no slower to set up than hyperfine (which I love for CLI benching): https://k6.io/

[jiehong]

Could hyperfine running curl be an alternative?

[mosselman]

That is what I do in my blog post.