[jonathanpoulter]

This seems like such a significant problem for the language and has been around since the beginning of time. Considering the amount of value that Python adds to companies and individuals around the world, is there a reason that institutions or someone with the means hasn't funded a project to "solve" the GIL problem?

[mattip]

PyPy put up a prototype and proposal to deliver a GILless python for $100,000. No one took us up on the offer. It is still open.

https://morepypy.blogspot.com/2017/08/lets-remove-global-int...

[jashmatthews]

I find it really sad that no company will fund this. One of the huge python deployments would recoup this in hosting quickly.

Instead we’re all just burning more money on AWS.

[acdha]

It's far more common to have Python programs be I/O bound, and when they are CPU bound it's often not due to GIL contention (remember that C extensions can drop the GIL before doing something lengthy). It would be nice if the GIL was gone but a fair fraction of Python developers would not notice much difference.

[pdonis]

> It's far more common to have Python programs be I/O bound

But part of the reason for that is that Python programmers know there's no point in trying to run multiple CPU-bound tasks in the same Python process, so they don't try.

> C extensions can drop the GIL before doing something lengthy

Yes, but they're still limited in what they can do--as soon as they call back into a Python bytecode they're GIL-bound again. And if you depend on C extensions any time you need CPU bound concurrent tasks, you're giving up a lot of the advantages of using Python in the first place.

[acdha]

It’s part of the reason but not all or, I suspect, even most: the number of things which people need to compute with multi threading but without much I/O is not an especially large fraction of what people use pure Python for. If you need raw CPU speed for arbitrary code, it’s not the language most people would pick.

The exceptions also tend to have existing high-quality extensions (crypto, compression, image processing, etc.) so while it’s technically true that you’re giving up Python most people aren’t doing that personally - they’re just calling Pillow or numpy - or they’re using it for a tiny fraction of the total program.

Frequently this ends up being the same speed or even faster than using other languages because most people are either using the same C libraries or learning just how many optimizations their simple implementation lacked.

Again, it’s not hard to come up with things where the GIL is inarguably a bottleneck but it comes up a lot more in debates than real-life in my experience.

[andreareina]

I'm one of those with embarrassingly-parallel cpu-bound workloads. The multiprocessing module works, but the extra bookkeeping and plumbing over an actually-parallel-multithreading implementation is a pain in the butt. That said, the multithreading speedup is both faster and easier than porting to another language.

[acdha]

Definitely - I used to support a computational lab and have worked on enough other CPU-bound problems to have plenty of things which I wouldn’t recommend Python for. I just think that as a field we’re predisposed to focus on that as the kind of work Real Programmers™️ do when most groups are limited by their ability to implement and maintain business logic long before they hit the wall on what Python can do.

[nateberkopec]

Does I/O release the GIL in Python?

[pansa2]

Yes, generally.

[snissn]

I think it's just not that big of a problem / can be worked around. I had a multi terabyte of RAM across 1000s of cores python product and we just used one process per core

[davvid]

Google threw money at the problem and that didn't help. Lookup the history behind the "Unladen Swallow" project.

[pansa2]

> Google threw money at the problem

Not really, especially compared to the amount of money they threw at JavaScript. IIRC Unladen Swallow was just a couple of interns.

[jashmatthews]

Unladen Swallow wasn’t actually a serious attempt like V8 or anything and a large part of why it failed was that LLVM wasn’t ready and even when it was, it was ill suited to compiling a highly dynamic language like Python.

[andreareina]

I think the big problem is all the code out there that depends on the GIL. A hypothetical python4 that removed the GIL would probably cause issues of the same general nature as the 2 -> 3 migration, though at a reduced magnitude.