| I use python in a scientific context, but have so far not written much extensions for python in any of the languages tested. I'm interested in some guidance which language a) is easy to integrate with python and b) has some good performance, but this benchmark lacks the details to come to any conclusion. I tried to run the benchmark on my own computer, but the setup documentation was not enough for me to get the julia integration running. I haven't used julia before, so it might just be something very simple. Similar I haven't used poetry much before, and the given documentation failed to install the necessary setuptools-rust for me. I could fix it on my own, but doesn't make me feel certain about the outcome of the benchmark. The rust benchmark did not reproduce for me: "Rust (Pyo3) parallel after" has a 1.56 speedup for me, but a 2.6x slowdown for the author. Also I don't understand what the difference between "after" and "before" is, the code just calls the same code twice. Might be a JIT/Cache thing, but it's unclear to me. One sentence what before/after refers to would be very helpful. Generally all measurements are only done once. Measuring at least thrice gives one at least a chance to detect an outlier and gives possibility for statistics, e.g. is a difference betwee the different cython annotations even meanginful? The "C Cython (pure-python mode)" is reported faster then "C Cython (.pyx)". The Cython project itself says that using pyx files should be faster, so something strange is going on. "Cython is fast, but none of these methods are able to release the GIL. " (A) this is not true (B) this seems to be mostly over single threaded performance, so why is that meaningful? "Rust is not that fast beacuse it needs to copy data; using Pyo3 objects would probably lead to similar results as cython, but with an added library." The rust code already contains Pyo3, so an "added library" is not necessary as far as I understand. I'd guess the performance stems more from conversions between different types then anything else. Maybe Julia (and the python-julia bridge) is particular smart about it and thus it's super easy to use, while pyo3 (and cython) needs some more work to interface with python. Even if that is true, I couldn't say it from the presented data. With these caveats resolved I'd be interested in the benchmark, but without it I can't really say anything from it. |
1) Calling into an actual static language like C or Rust is the best option. You get maximal performance and all the benefits of the static language. The downside is that you need to learn another language, and manage both languages in your project, including setup and compilation of the static language etc.
2) Cython is easiest for small-scale projects, since it integrates very well with Python, and you can learn it incrementally. But I found it annoying to work with - it felt like half a language that fell between Python and a proper static language. I ended up using Cython in the end, but I wasn't happy with it.
3) Numba looks interesting and promising. At least 2 years ago, it was too brittle and had too many situations where it didn't work or didn't give noticable speedups. I'm sure they improved it since then. I would definitely take a look.
You can always just learn Julia of course and have this entire problem of "my high-level language is too slow" completely disappear ;)