[nkurz]

I've been particularly bothered by the way that almost all "real numbers" in CS papers attempt to explain the advantages of one algorithm over another based on small performance differences of two partially optimized implementations, compiled with a single compiler with arbitrary options, and then tested on a single older machine. The few that go further merely repeat the exercise with an even older compiler an an even older machine with a forgotten architecture. While it's a step up from simply counting pseudo-instructions and memory accesses, and while there is room for theoretical analysis of asymptotic behaviour, it's painful how much is read into the particular tea leaves of one or two data points.