|
|
|
|
|
by girfan
1404 days ago
|
|
This is a great analysis; thanks for writing. I have also been working on running multiple Python interpreters in the same process by isolating them in different namespaces using `dlmopen` [1].
The objective on a high level is to receive requests for some compute intensive operations from a TCP/HTTP server and dispatch them on to different workers. In this case, a thin C++ shim receives the requests and dispatches them on to one of the Python interpreters in a namespace. This eliminates contention for the GIL amongst the interpreters and can exploit parallelism by running each interpreter on a different set of cores. The data obtained from the request does not need to be copied into the interpreter because everything is in the same address space; similarly the output produced by the Python interpreter is also just passed back without any copies to the server. [1] https://www.man7.org/linux/man-pages/man3/dlmopen.3.html |
|