Hacker News new | ask | show | jobs
by Znafon 1030 days ago
> I would be fine with have it never be removed, and I think we should try sub-interpreter first.

There is a lot of work going on with sub-interpreters and the per-interpreter GIL is shipped in Python 3.12.

The results are very impressive: https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/, as good as one could have hoped I think.

It seems to me like the work on sub-interpreters will continue in parallel ;) to the work on free-threading.

Sub-interpreters and no-GIL have different use-cases though.
4 comments
BiteCode_dev 1030 days ago
For now there is nothing impressive with it. The perfs they show off don't include data sharing, which is very primitive.

Which means there is nothing subinterpreters do that can't be done with multiprocessing.

They show progress, and it's good. But I would wait until we can see gunicorn spawning WSGI interpreters workers and get similar performances to the setup with regular workers to get enthusiastic.

link
sidewndr46 1030 days ago
to further your point: I've used the technique launching a separate python process for performance critical operations for years on different projects. I can run it in a completely different priority set as well.
link
jnwatson 1030 days ago
Once no-GIL is integrated, what is the use case for sub-interpreters?
link
kodablah 1030 days ago
I have a use case for sandboxing (without a secure-sandboxing requirement) that it would help solve. But at last read, they discourage/disallow sharing modules across the subinterpreters and reloading all modules in each subinterpreter is not acceptable for my use case.
link
scruple 1030 days ago
That's very interesting, thank you for the link. I've used Python since the 00s and I don't ever recall having come across sub-interpreters...
link
Znafon 1030 days ago
They were always usable using the C-API but not exposed to the Python library (and until 3.12 you were blocked by the shared GIL so there was little point anyway).

Now that the work from Eric Snow has been merged, you can use https://pypi.org/project/interpreters-3-12/ to create one from Python code.

link
ed_blackburn 1030 days ago
I'd be interested in introducing appropriate abstractions (ABI) and in a module for parallelism. Whereby one can swap out different implementations for sub-interpreters, green threads, GIL or even hardware or cloud vendor-optimised implementations. Not that dissimilar to Project Loom in Java.
link