| > This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general. Statically linking everything doesn't work in this case because each C extension module (i.e. each Python package with some native code in it) is, in effect, a shared library. If you statically linked the C++ protobuf library into both the Python protobuf library and the Python tensorflow library then you'd end up with two copies of the library in the process (with two copies of globals/statics such as the protobuf global descriptor pool). That's not a problem in itself, but it wouldn't be possible to pass protobuf objects between them. (In principle, you could perhaps statically link protobuf into the CPython executable, but I really don't think Google is doing that.) The usual solution is the exact opposite to static linking: you have to build the C++ protobuf library as a shared library (libprotobuf.so) and then link to it from both of your C extension modules (i.e. from the other two shared libraries). Following the tiny crumbs of evidence of how to do this for protobuf, it seems like that was previously how you would do it. But it's possibly not now? It seems like proto_api.h [1] has a way to get the global descriptor pool, possibly to work around needing a common shared library? But also now that header is deprecated anyway [2]? (I think this is roughly where I got to before when I gave up.) [1] https://github.com/protocolbuffers/protobuf/blob/main/python... [2] https://github.com/protocolbuffers/protobuf/issues/9464 |
Yes, we actually do exactly this (not just protobuf but generally all of the other C code with it for the given application). But there are other approaches that can work.
> But also now that header is deprecated anyway
It's deprecated exactly because the preferred/supported/default runtime is the upb based runtime. The odd gencode design is what made that transition possible as an implementation detail. So the deprecation is just the same topic here, that people successfully setting up the dynamic linking is not really realistic/viable unfortunately, and the effort for open source support is spent making the best behavior on the upb runtime which can't have that feature.