[mikemike]

<rant>

Oh, well ...

The Lua port of Richards I've seen is naive and nobody within his right mind would write Lua code like that. As has been pointed out elsewhere, Richards is a very atypical benchmark and different implementation styles have very different performance characteristics.

In fact, PyPy wins ONLY on Richards by 1.2x and ONLY for a specific iteration count. But LuaJIT wins substantially on ALL OTHER BENCHMARKS where something is actually compiled (i.e. not regex stuff). Your paper clearly shows that in the appendix, but no matter what, you're still pointing out that one little result.

And, even worse, you're trying to draw conclusions from this outlier. I have absolutely no idea how you could possibly come to the conclusion that this makes PyPy more suitable for 'large scale' programs? And that LuaJIT is not? That's not just unscientific -- it's dishonest and disparaging.

Also, constantly repeating that 'hand-crafted JIT compilers' are somehow inferior to PyPy's overly complex approach is getting really old by now. LuaJIT runs on more architectures and more platforms than PyPy ever will. And it's vastly ahead in terms of optimizations. Now, please compare the timeframe all of this was implemented and the size of our teams. Ok, so what was the better deal, again?

And, no, the semantics of Lua and Python are not that far apart as you'd like everyone to believe. Dispatch is more complex in Python, but that can be taken care of in a central place in the code. Most library code can be written in terms of lower-level code. And so on. There are simple solutions for all of that. And none of that warrants a clumsy meta-tracing approach.

</rant>

[fijal]

Hi Mike.

Seriously, I would not try to say LuaJIT is worse than PyPy, ever. I picked richards, because it's the best benchmark from the set I know, while still being not a very good one. I'm all for saying "this is a bad benchmark".

I think you're putting words into my mouth. I did not say PyPy is better for large scale programs, I said we have more focus and I derive it from what we use as PyPy benchmarks. Since it's impossible to compare, because we lack programs we don't actually know, but my gut feeling would be that LuaJIT would be slightly faster on something large.

The problem with Python (and why we choose our approach instead of hand crafting the JIT) is that Python is a very large language, with lots of builtin modules and types interfering in very unobvious ways, while Lua is not. This is why we decided it makes more sense to go that way, we probably wouldn't do it for Lua. People tried to write a JIT by hand for Python and failed so far, I'm however fine with the explanation we're just too dumb to do that.

Overall I find your answer way too agressive - I didn't claim PyPy is better, I mostly pointed out to things I know and added it's very unscientific and that scientific approach is very hard to get. Also your attack on our meta-tracing is unfounded - Python is just a very very large language with strange corner cases and we have to somehow deal with it. Feel free to implement it differently, but I'm happy with our approach.

[haberman]

You did say that LuaJIT "usually cares for smaller stuff," implying that it isn't optimal (or as optimal as PyPy) for larger stuff. I could see how someone who had worked for years to optimize LuaJIT for all cases could find that disparaging. It sounds like you're writing off LuaJIT as only meant for toy programs.

[fijal]

Well. the luaJIT benchmark page does not contain any large benchmarks, while it does contain computer language shootout ones which are small. There might be many many legitimate reasons for that, but it does imply that LuaJIT cares for small stuff more, since all the public benchmarks are small.

[ltratt]

I know myself that I had a high regard for LuaJIT before benchmarking, and an even higher regard for it afterwards. Richards is an oldish benchmark, and none of the subsequent versions (Lua, Python, Converge) is exactly "native", so I'm not sure that any of those languages has an advantage. It's a datapoint, no more, no less. As to which VM would be faster on "large" programs - well, personally I agree with your implication that it's a matter of conjecture, and we'll probably never know because it's hard to write large cross-language benchmarks.

I do, however, think you might be a little harsh on meta-tracing. It's an interesting approach, in no small part because it allows VMs to be written for languages other than Python. LuaJIT can be retargeted too (I've been impressed by Thomas Schilling's work), but I think it would be fair to say that it's harder work than writing a new RPython interpreter. In other words, IMHO, there is plenty of scope for both LuaJIT and RPython/PyPy to co-exist happily alongside each other.