The GIL does not prevent race conditions in your Python code. It only prevents race conditions in internal data structures inside the interpreter and in atomic operations, i.e., operations that take a single Python bytecode. But many things that appear atomic in Python code take more than one Python bytecode. The GIL gives you no protection if you do such operations in multiple threads on the same object.
I think what many will experience, is that they want to switch to multithreading without GIL, but learn they have code that will have race conditions. But that don't have race conditions today, as it's run as multiple processes, and not threads.
For instance our webserver. It uses multiple processes. Each request then can modify some global variable, use as cache or whatever, and only after it's completely done handling the request the same process will serve a new request. But when people see the GIL is gone, they probably would like to start using it. Can handle more requests without spamming processes / using beefy computer with lots of RAM etc.
And then one might discover new race conditions one never really had before.
If you are writing Python code, the GIL can already be dropped at pretty much any point and there isn't much way of controlling when. Iirc, this includes in the middle of things like +=. There are some operations that Python defines as atomic, but, as I recall, there aren't all that many.
In what way is the GIL preventing races for your use case?
It is not about your code, it is about C extensions you are relying on. Without GIL, you can't even be sure that refcounting works reliably. Bugs in C extensions are always possible. No GIL makes them more likely. Even if you are not the author of C extension, you have to debug the consequences.
I mean that, if the GIL didn't prevent races, it would be trivially removable. Races that are already there in people's Python code have probably been debugged (or at least they are tolerated), so there are some races that will happen when the GIL is removed, and they will be a surprise.
The GIL prevents the corruption of Pythons internal structures. It's hard to remove because:
1. Lots of extensions, which can control when they release the GIL unlike regular Python code, depend on it
2. Removing the GIL requires some sort of other mechanism to protect internal Python stuff
3. But for a long time, such a mechanism was resisted by th Python team because all attempts to remove the GIL either made single threaded code slower or were considered too complicated.
But, as far as I understand, the GIL does somewhere between nothing and very little to prevent races in pure Python code. And, my rough understanding, is that removing the GIL isn't expected to really impact pure Python code.
My understanding, is that many extensions will release the GIL when doing anything expensive. So, if you are doing CPU or IO bound operations in an extension _and_ you are calling that operation in multiple threads, even with the GIL you can potentially fully utilize all of the CPUs in your machine.
> removing the GIL isn't expected to really impact pure Python code.
If your Python code assumes it's just going to run in a single thread now, and it is run in a single thread without the GIL, yes, removing the GIL will make no difference.
> If your Python code assumes it's just going to run in a single thread now, and it is run in a single thread without the GIL, yes, removing the GIL will make no difference.
I'm not sure I understand your point.
Yes, singled thread code will run the same with or without the GIL.
My understanding, was that multi-threaded pure-Python code would also run more or less the same without the GIL. In that, removing the GIL won't introduce races into pure-Python code that is already race free with the GIL. (and that relatedly, pure-Python code that suffers from races without the GIL also already suffers from them with the GIL)
Are you saying that you expect that pure-Python code will be significantly impacted by the removal of the GIL? If so, I'd love to learn more.
> if the GIL didn't prevent races, it would be trivially removable
Nobody is saying the GIL doesn't prevent races at all. We are saying that the GIL does not prevent races in your Python code. It's not "trivially removable" because it does prevent races in the interpreter's internal data structures and in operations that are done in a single Python bytecode, and there are a lot of possible races in those places.
Also, perhaps you haven't considered the fact that Python provides tools such as mutexes, locks, and semaphores to help you prevent races in your Python code. Python programmers who do write multi-threaded Python code (for example, code where threads spend most of their time waiting on I/O, which releases the GIL and allows other threads to run) do have to use these tools. Why? Because the GIL by itself does not prevent races in your Python code. You have to do it, just as you do with multi-threaded code in any language.
> Races that are already there in people's Python code have probably been debugged
Um, no, they haven't, because they've never been exposed to multi-threading. Most people's Python code is not written to be thread-safe, so it can't safely be parallelized as it is, GIL or no GIL.