Hacker News new | ask | show | jobs
by jupp0r 1369 days ago
Static modules are loaded as shared libraries/dlls. The way operating systems implement this is that each library is loaded once per process and that statically allocated memory is mapped into the virtual address space of the process. You can't load one so/dll multiple times in some sort of container, so each module would have to implement this isolation inside their module, probably through some sort of API that the python runtime offers to the module. It's not rocket science but it will definitely break existing code where it's common practice to use dll lifetime hooks as initialization code that allocated some global state that's conveniently shared throughout the module.
1 comments
girfan 1368 days ago
> You can't load one so/dll multiple times in some sort of container

I believe you can do that with `dlmopen` in separate link maps. I have worked with multiple completely isolated Python interpreters in the same process that do not share a GIL using that approach.

link
unnah 1368 days ago
Thank you for the hint about dlmopen! I had a problem that can be solved by loading multiple copies of a DLL, and it looks like reading manpages of the dynamic linker would have been a better approach than googling with the wrong keywords.
link
girfan 1367 days ago
That's great!

There are a few cases where `dlmopen` has issues, for example, some libraries are written with the assumption that there will only be one of them in the process (their use of globals/thread local variables etc.) which may result in conflicts across namespaces.

Specifically, `libpthread` has one such issue [1] where `pthread_key_create` will create duplicate keys in separate namespaces. But these keys are later used to index into `THREAD_SELF->specific_1stblock` which is shared between all namespaces, which can cause all sorts of weird issues.

There is a (relatively old, unmerged) patch to glibc where you can specify some libraries to be shared across namespaces [2].

[1]: https://sourceware.org/bugzilla/show_bug.cgi?id=24776#c13

[2]: https://patchwork.ozlabs.org/project/glibc/patch/20211010163...

link
jupp0r 1368 days ago
IIRC glibc is limited to 16 namespaces though.
link
girfan 1367 days ago
Currently it is, yes. I am not sure how fundamental it is. I tried patching glibc to support more (128 in my case) and it seemed to work fine.
link