[rogerbinns]

Note that free threaded compatible doesn't necessarily mean the package supports free threading (concurrent execution), just that it can be loaded into a free threaded interpreter.

This is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GIL full and free threaded interpreters around. They have a different ABI so you can't use extensions compiled against one with the other.

Free threading is at the beginning of its journey. There is a *lot* of work to on all C code that works with Python objects, and the current documentation and tools are immature. It is especially the case that anyone doing Python concurrent object mutation can cause corruption and crashes if they try, and that more auditing and locking need to be done in the C code. Even modules in the standard library have only been partially updated.

You can see a lot details and discussion in the comments at https://news.ycombinator.com/item?id=45633311

[LtWorf]

Why do you provide it at all then if it's not working as intended yet?

[rogerbinns]

As I stated:

> so that you don't have to keep separate GIL full and free threaded interpreters around

It means the user doesn't have to keep two Pythons around, install packages in both of them, etc.

It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.

[LtWorf]

But the users install the free threaded python to do free threaded stuff. The second they use your package they have a GIL again, which entirely defeats the point.

Wouldn't it be much better to just not support it if it's not supported?

[rogerbinns]

What is better:

1) Saying your package supports free threading, but it isn't safe - ie concurrent mutation can result in corruption and crashes

2) Allowing the package to be loaded into a free threaded Python, which immediately enables the GIL. Concurrent mutation does not result in corruption and crashes because of the GIL. The user doesn't have to maintain two Python installations. They can set the environment variable PYTHON_GIL=0 or start Python with -Xgil=0 which will keep the GIL disabled, and they will be fine if they avoid concurrent mutation.

I chose 2. The stdlib json package (along with many others) picked 1. Heck I'll guarantee that most that picked 1 aren't 100% safe either, because doing the changes is hard work, *every* case has to be covered, and tools like thread sanitizers don't work.

The reason I chose 2 is because I care about data integrity. I will eventually reach 1, but only once I can be certain the code is correct.

[LtWorf]

3) Saying your package doesn't support free threading instead of adding a GIL and force the users to stick to regular python.

[rogerbinns]

You aren't forced to use a GIL as I keep stating. You can set an environment variable or a command line flag to Python and the GIL will remain disabled. My package will work just fine if you do that, unless you provide it with data you concurrently modify in which case you can get corruption and crashes.

[cyanydeez]

Do you atleast warn users? This sounds like madness.

[rogerbinns]

Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded.

Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far.

Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code.

But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.

[LtWorf]

I think what you are doing is hiding problems. I think crashes and bugs are preferable to find the issues at this point. People who want the safe option will run the regular python.

[rogerbinns]

I provide 3 options:

1) Use regular GIL Python and you get the highest levels of integrity and correctness of operation of my package

2) Use a free threaded Python, the GIL will be enabled at load time, and you get the highest levels of integrity and correctness

3) Use a free threaded Python, and set $PYTHON_GIL=0 or start with -Xgil=0 to keep the GIL disabled, and providing you do not do concurrent mutation of data provided to my package, you get the highest levels of integrity and correctness

BTW I did not randomly choose to provide the free threaded builds. I specifically asked the setuptools maintainers (under the Python Packaging Authority) how to prevent free threaded builds for PyPI. They encouraged me to do the free threaded builds so that a user doesn't to have maintain parallel regular and free threaded Python installations. And it allows option 3 above.