[lmm]

Sure, but with most AOT compliers you only get one machine-code version of your code, which is easier to benchmark than worrying about the multiple levels of a JIT.

[pjmlp]

Actually I think many developers aren't aware of the right tools for this type of job.

JITs are easy to benchmark when tools like Oracle Studio, Java Flight Recorder, dotTrace and so forth are used.

The problem is trying to benchmark them using old style concepts, e.g. outputting time differences to a text file.

[RussianCow]

Those tools don't exist for all JITed platforms. Hence the problem.