[mkaufmann]

Concerning your critique of the in your opinion overly hard scientific review process. According to her blog the paper was rejected because: "Reviewers at conferences [...] have been very skeptical, and pressed us multiple times to produce some kind of comparison of basic block versioning against tracing."[1] I think this is a very valid concern. When you describe a new scientific approach (Basic Block Versioning) you should compare to the state of the art. Otherwise it is very hard for the reader to judge the merit of the new approach. However I agree with your sentiment that a new approach should not only be judged by performance numbers (there is actually a nice article going deeper on this topic on databasearchitects[2]). But there should at least be a more thorough theoretical discussion than only 3-4 sentences. Benchmarks comparing to existing approaches in this case can help to show pathological cases which might indicate weaknesses of the approach or to empirically demonstrate the feasibility of the new approach.

[1] http://pointersgonewild.wordpress.com/2014/11/14/the-fastest...

[2] http://databasearchitects.blogspot.de/2014/09/experiments-hu...

EDIT: Many conferences also allow publishing papers without a deep comparison to exisiting research in the industrial session. This allows demonstrations of interesting implementation variants or system choices

EDIT 2: The review critisim in older blog posts "Conference reviewers criticized us for not discussing compilation times, and raised the issue that perhaps basic block versioning could drastically increase compilation times." is also very valid for a jit compiler. Again discussion does not have to mean that you have to be faster than all existing systems. Paper acceptance is always a little bit a random process, but at least in this case the review comments are valid from my point of view and her phd advisor should probably have detected these problems in proofreading before submitting. I really hope that the paper will finally be accepted!