Hacker News new | ask | show | jobs
by rcxdude 161 days ago
Stability and accuracy, when applied to clocks, are generally about dynamic range, i.e. how good is the scale with which you are measuring time. So if you're talking about nanoseconds across a long time period, seconds or longer, then yeah, you probably should care about your clock. But when you're measuring nanoseconds out of a millisecond or microsecond, it really doesn't matter that much and you're going to be OK with the average crystal oscillator in a PC. (and if you're measuring a 10% difference like in the article, you're going to be fine with a mechanical clock as your reference if you can do the operation a billion times in a row).
1 comments
jacquesm 161 days ago
This setup is a user space program on a machine that is not exclusively dedicated to the test running all kinds of interrupts (and other tasks) left, right and center through the software under test.
link
loeg 161 days ago
For something like this, you can just take several trials and look at the minimum observed time, which is when there will have been ~no interruptions.

https://github.com/facebook/folly/blob/main/folly/docs/Bench...

link
jacquesm 161 days ago
You don't actually know that for sure. You have only placed a new upper bound.
link
loeg 161 days ago
This seems like more of a philosophical argument than a practical one.
link
jacquesm 161 days ago
No, it is a very practical one and I'm actually surprised that you don't see it that way. Benchmarking is hard, and if you don't understand the basics then you can easily measure nonsense.
link
jerrinot 161 days ago
You raise a fair point about the percentiles. Those are reported as point estimates without confidence intervals and the implied precision overstates what system clock can deliver.

The mean does get proper statistical treatment (t-distribution confidence interval), but you're right that JMH doesn't compute confidence intervals for percentiles. Reporting p0.00 with three significant figures is ... optimistic.

That said I think the core finding survives this critique. The improvement shows up consistently across ~11 million samples at every percentile from p0.50 through p0.999.

link