[xeromal]

Coming from C#, I honestly HATE python's multiprocessing and multithreading. Hell, I hate it's async await. I learned recently that in one mode, it pipes the values across the process and this made it impossible to use when passing along large pandas dataframes. I'm sure half of it is just my own lack of knowledge with python's abilities but C# sure made it easier. lol

[nerdponx]

For Pandas, I recommend the third-party Joblib library: https://joblib.readthedocs.io/en/latest/

[paddy_m]

That looks a bit low level. I would look at dask and polars. Dask scales to multiple processes on a single machine and to multiple machines and its dataframe looks pretty close pandas. Polars uses multiple cores on the same machine better than pandas (not sure about dask), but has a significantly different dataframe api than pandas. Polars, primarily through lazyframes enables much higher single core performance too.

[nerdponx]

Yeah, it's "low level" in the sense that you still have to manually chunk up your data. I agree that Dask, Polars, etc are better if you want a more transparent distributed computing experience. Joblib is great for if you already have working single-process code and you just want to parallelize it. It's what Scikit Learn uses internally, for example.

But as it pertains to the original thread topic, it's still fairly high-level. I'd consider it bit higher-level than concurrent.futures for example.