|
|
|
|
|
by dekhn
1061 days ago
|
|
Over quite some time I've become convinced multiprocessing module is better than an optional GIL removal. It may leave many useful bits on the table (compared to pure multithreaded coding, like C++/pthreads) but I've still been able to get it to scale my application performance (CPU-bound, large-memory) to the number of cores of even large boxes (96+ vCPUs). IIRC the future/concurrent library was key to being productive. 20 years ago I would said different, as at the time, IronPython demonstrated a real alternative to CPython that was faster, and fully multitrhreaded (including the container classes). |
|
Only problem is that 99% of that CPU usage is for serializing/deserializing IPC messages and total throughput would have been higher using a single process.
There are use-cases for multiprocessing. As long as data sharing between processes is insignificant, it can be quite performant. Just like using a bash-wrapper script that orchestrates a bunch of python (or other) processes.