[hedora]

Their first iteration is an entire run of the underlying benchmark. Subsequent iterations are reusing the same VM. They run each plot multiple times, and reboot between plots.

They’re trying to show that “warmed up steady state” isn’t something that reliably exists.

[pso]

Yes, I know. But the tone of the whole article, is as if, they've found deep flaws across many VMs. They call something 'warmup" which I think has little or nothing to do with the JIT, but is unaccounted variations in the whole running system.

The final graph shows a binary trees program in C, with a 6% variation between "in process executions", and no steady state, it seems logical that most VMs will show the same or worse variation.

The "warmed-up steady state" does exist, but not if they define it so narrowly. All of their iterations and timings are running at x30 to x100 interpreted speed, the only 'cold' interpreted code is in a few microseconds of the first loops of an execution.