[mattip]

Third party libraries like SciPy scikit-learn, pandas, tensorflow and pytorch have been critical to python’s success. Since CPython is written in C and exposes a nice C API, those libraries can leverage it to quickly move from (slow) python to (fast) C/C++, hitting an optimum between speed of development and speed of runtime.

PyPy’s alternative, CFFI, was not attractive enough for the big players to adopt. And HPy, another alternative that would have played better with Cython and friends came too late in the game, by that time PyPy development had lost momentum.

[toxik]

PyPy on numpy heavy code is often a lot slower than CPython

[mattip]

Yes. The C API those libraries use is a good fit to CPython, a bad fit to PyPy. Hence CFFI and HPy. Actually, many if the lessons from HPy are making their way into CPython since their JIT and speedups face the same problems as PyPy. See https://github.com/py-ni

[jjgreen]

I rather like Python and have used the C API extensively, "nice" is not the word I'd choose ...

[glkindlmann]

Sorry can you explain more the connection between PyPy and CFFI (which generates compiled extension modules to wrap an existing C library)? I have never used PyPy, but I use CFFI all the time (to wrap C libraries unrelated to Python so that I can use them from Python)

[mattip]

CFFI is fast on PyPy. The JIT still cannot peer into the compiled C/C++ code, but it can generate efficient interface code since there is a dedicated _cffi_backend module built into PyPy. Originally that was the motivation for the PyPy developers to create CFFI.

[glkindlmann]

Thank you for the background info, and sorry for me explaining CFFI (I just wanted to be sure we were talking about the same thing). Being ignorant about PyPy, I honestly had no idea until now that there was a personnel or purpose overlap between CFFI and PyPy. I am very grateful for CFFI (though I only use it API mode).

[pjmlp]

Python was already widely deployed before them, thanks to Zope, and being a saner alternative to Perl.