[pookha]

Gotten burned by that. Have a python utility that calls something that's very CPU intensive (GDAL) and I literally have to wrap this python process in java for real concurrency. Not a python expert by any means but I'd love to know what other people do when this happens. I'm assuming it's something revolving around containers and messaging (blech)...

[antod]

I thought Python calling a native lib was the one time you could multithread your way around the GIL if you were cpu limited? Are the GDAL bindings not releasing the lock or something?

[ska]

Is your problem that the c part of GDAL isn't concurrent and you are calling it multiple times, or are you blocking on something you shouldn't?

Python calling a concurrent c thing is super common. For more complex things there is multiprocessing, but that essentially is simplification of IPC.

[pookha]

Had a massive amount of images that I was trying to process for a specific purpose. I wound up spinning up multiple python processes within a java wrapper using concurrent linked queues. Got the job done but it's not the proudest moment of my life.

[seunosewa]

Why did you need the Java wrapper? You can orchestrate stuff like that within Python. The multiprocessing module.

[giantrobot]

You could have done this with the multiprocessing module. It interfaces like the threading library but it's launching new processes for each "thread". I've done a shitload of image processing in Python with it. You'll end up with a 100% Python version of what you did.

[ska]

Multiprocessing does basically the same thing, but in python.

[totony]

Python works around this by having multithreading through multiprocessing and IPC. its made pretty easy to use with the futures builtin module but is finicky/slow/hard to support.