Hacker News new | ask | show | jobs
by dekhn 1061 days ago
Over quite some time I've become convinced multiprocessing module is better than an optional GIL removal.

It may leave many useful bits on the table (compared to pure multithreaded coding, like C++/pthreads) but I've still been able to get it to scale my application performance (CPU-bound, large-memory) to the number of cores of even large boxes (96+ vCPUs). IIRC the future/concurrent library was key to being productive.

20 years ago I would said different, as at the time, IronPython demonstrated a real alternative to CPython that was faster, and fully multitrhreaded (including the container classes).

2 comments

Sure, with multiprocessing you can get 96 python processes running at 100% CPU while sharing a large dataset.

Only problem is that 99% of that CPU usage is for serializing/deserializing IPC messages and total throughput would have been higher using a single process.

There are use-cases for multiprocessing. As long as data sharing between processes is insignificant, it can be quite performant. Just like using a bash-wrapper script that orchestrates a bunch of python (or other) processes.

Whatever happened to ironpython? I used to do a lot of C# development and remember dabbling with ironpython back in the day. It seemed like it was important to Microsoft, .Net added the whole concept of dynamic data types mostly to support ironpython and ironruby. But I never really used python much until recently, so of course when I finally needed to do python I looked for ironpython and it doesn’t appear to be a thing anymore.
It looks like Microsoft abandoned these dynamic language implementations in 2010. Maintaining parallel implementations of two complex, mature scripting languages is a huge feat. It would take some very expensive talent. That said, IronPython was loved by those who used it, which means it captured them in the DotNet ecosystem. Perhaps that win was not enough for Microsoft to continue the project. Ideally, Python foundation should "own" (and fund) Jython and IronPython development, but that takes (a lot of) money. (Sorry, I'm much less familiar with Ruby and IronRuby.)
It is still a thing, but it's open source now instead of maintained by Microsoft. There was a release that finally supports Python 3 in December last year.

I don't know how useful it is really, if you really want performance then you probably shouldn't choose Python to begin with, or you use the libraries which may not be compatible with IronPython. These days it barely takes me longer to build a simple script in C# than in Python either.

It's so so. Pythons core value is it's huge stack of lib's. And most important fall down with IP due to them using c and so on.

When we needed python c# interop it was better to use python.net and integrate that way. Annoying to setup but when it works you can get both to work seamlessly