[twa927]

I was trying to speed up a log processing service running on PyPy by rewriting it in Java. I was surprised that the result was about twice slower (I know Java quite well and I didn't see obvious optimizations; most of the time was spent in GC). So it can be quite fast even in more absolute terms (VM languages), at least for some types of code.

[spullara]

If more than 1% of your time is in GC you are doing something very wrong.

[michaelmrose]

The fact that a singular implementation was better than java says less about the languages and more about the particular software.

[tigershark]

I know that it’s asking a lot, but any chance that you can post a minimum reproducible sample? From what I know it is quite smelly...

[twa927]

I don't have access to this codebase now but I'll try to write some benchmark.

[tigershark]

If you manage to do it would be awesome, otherwise thanks a lot anyway for the effort :) I’m just curious to understand why it happens because it’s exactly the opposite of what I would expect. The only explanation that comes to my mind is excessive gc as someone else already mentioned, but it would be interesting to see the original code.

[twa927]

I started doing it but yes, it's too much effort to get two full benchmarks, I'm sorry :). But I think it went down to the inefficiency of String.split(): https://stackoverflow.com/questions/37007189/string-split-te... and generally the Java's String built-in methods not being GC-friendly: https://stackoverflow.com/questions/20336459/garbage-friendl.... I'm guessing that when such parts can be coded in a non-VM environment (CPython/PyPy runtime) they can be made much faster, and Java to these days has the standard library coded in pure Java?

[tigershark]

Yes, Java has nothing like C# Span to avoid these kind of problems, but I thought that also python would be affected in a similar way... Anyway, thanks for sharing.

[auscompgeek]

CPython and PyPy are both VMs, similarly to the JVM.

[twa927]

I meant the inside of a VM (its implementation), which is coded in C/RPython. Java's VM is coded in C++, but I don't think C++ is used for any regular library functions, while C is used heavily for Python's stdlib.

[d0mine]

I would have expected it: https://stackoverflow.com/questions/9371238/why-is-reading-l...

[int_19h]

I had a binary parser written in Python that took around 30 seconds on typical input on CPython. PyPy took that down to about 10 seconds. Rewriting it in C# took it down to 200 ms.

[twa927]

If this was using a loop processing a single byte in an iteration I would expect a greater speedup on PyPy. I've seen 100x speedup in such cases.

[int_19h]

Not single byte, but individual fields (float32/int32/string etc). Yes, I expected a much more significant speed-up as well. It's probably because a lot of that code was driven by reflection-type techniques.

Curiously, IronPython did better than anything (but still slow). Haven't tried Jython.

Compiling the whole thing with Cython was less effective than PyPy.

[twa927]

Yes, reflection voids many JIT paths in PyPy, AFAIK. Maybe it was worth rewriting the Python code to get rid of the reflection?