[esrauch]

Engineer who works on Google Protobuf here, commenting as myself and not as an official statement.

It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable with any compatible thing that can understand Protobuf encoding including this one.

Kudos to Buf for making something that I'm sure a ton of people will find useful and which takes conformance so seriously.

Just to chime in with some context about Google's own implementations here though (since that's a lot of the discussion otherwise).

Google definitely takes Protobuf seriously including for the long term: you can't really understand how engrained it is within the Google stack without seeing it for yourself. It's not just RPC layer, it's storage, logging, FFI. Html templating is driven off Protobuf messages. Systems which interact with bank XML based systems uses Protobuf schemas. Internally it's widely used for in-memory library api types even without any direct/obvious connection to serialization just because it makes internal details like logging easier. This extremely large surface does create constraints and use-cases to balance. You can see Buf's reported numbers reflect that it is faster for a usecase they expect is typical, but at scale users do fall into the other buckets shown, affecting the performance of preexisting code is a major concern for our implementations that a greenfield implementation doesn't have.

Wide exposure in critical paths alongside long term support directly causes some quirks: for example some of our APIs followed PEP8 when it was created but PEP8 changed. It looks stupid that we have wrong style APIs but also it would be stupider to break compatibility for style reasons. JavaProto as another example still supports Java8 and the runtime is compatible with 2014 gencode which is a pretty major constraint.

Google Py Proto implementation has one extra interesting choice of the same gencode is reused with 3 different implementations (upb, a complete pure python one, and one that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++), which is why design the way that it is with runtime created classes, the pyi files are readable but the .py files not.

This definitely has pros and cons, and the direct approach taken by Buf here really makes a ton of sense. It's just that Google's maintained implementation falls into a different spot in a larger technical tradeoff space.

If you see things that appear to make no sense with the official implementations, feel free to file an issue on GitHub and we can look, sometimes there is no reason and we can fix it, and sometimes there's a reason which we can explain.

Kudos again to Buf here, I'm fully sure this will solve some set of real business needs better than Google's (but not because Google isn't maintaining our offerings too).

[quietbritishjim]

> ... that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++

This is a valid explanation for the odd implementation of the Python protobuf library (the only convincing one I've heard). But, the last time I looked into it (just a couple of years ago), it didn't seem reasonably possible to make use of this.

I can't remember the exact issue I think maybe the C++ library would be statically linked into the Python C extension module, which made it virtually impossible to use it from your own C++ code. Or maybe the issue was just that there was no C++ version prebuilt on pypi (the default is the upb version).

Anyway, it seems a pity to go to such enormous lengths for one feature and then make it essentially unavailable.

[esrauch]

Well, there's still two implementations even without getting into the quoted case. But yeah, narrowly about the C++Proto shared memory thing, unfortunately it's so hard to make this work for many different reasons that we don't even advertise support for this functionality at all. This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general.

But its kind of a clear example of a surprisingly complicated technical space. The big picture is that it's actually not weird when a simpler thing which has less constraints can be better on some axes users when compared to a complicated thing that delivers on many advanced constraints.

If "readability of the .py gencode instead of the .pyi gencode" was the worst pain point (or pet peeve) here then I actually suspect Buf wouldn't have even bothered, it's just one of these that is easy to see and explain.

[quietbritishjim]

> This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general.

Statically linking everything doesn't work in this case because each C extension module (i.e. each Python package with some native code in it) is, in effect, a shared library.

If you statically linked the C++ protobuf library into both the Python protobuf library and the Python tensorflow library then you'd end up with two copies of the library in the process (with two copies of globals/statics such as the protobuf global descriptor pool). That's not a problem in itself, but it wouldn't be possible to pass protobuf objects between them.

(In principle, you could perhaps statically link protobuf into the CPython executable, but I really don't think Google is doing that.)

The usual solution is the exact opposite to static linking: you have to build the C++ protobuf library as a shared library (libprotobuf.so) and then link to it from both of your C extension modules (i.e. from the other two shared libraries).

Following the tiny crumbs of evidence of how to do this for protobuf, it seems like that was previously how you would do it. But it's possibly not now? It seems like proto_api.h [1] has a way to get the global descriptor pool, possibly to work around needing a common shared library? But also now that header is deprecated anyway [2]? (I think this is roughly where I got to before when I gave up.)

[1] https://github.com/protocolbuffers/protobuf/blob/main/python...

[2] https://github.com/protocolbuffers/protobuf/issues/9464

[esrauch]

> principle, you could perhaps statically link protobuf into the CPython executable, but I really don't think Google is doing that.)

Yes, we actually do exactly this (not just protobuf but generally all of the other C code with it for the given application). But there are other approaches that can work.

> But also now that header is deprecated anyway

It's deprecated exactly because the preferred/supported/default runtime is the upb based runtime. The odd gencode design is what made that transition possible as an implementation detail. So the deprecation is just the same topic here, that people successfully setting up the dynamic linking is not really realistic/viable unfortunately, and the effort for open source support is spent making the best behavior on the upb runtime which can't have that feature.

[bigcityslider]

The regular proto-to-python feels fine to me. I quit Google a while ago and have still been using protobufs, and the main things that usually make teammates wary are: 1. No way to load .proto specs at runtime (there are GH issues about this) 2. No clear and easy way to use it with HTTP/1.1. Neither is specific to Python.

Both of these might sound silly if you're used to protobufs, cause you can build little helpers for both, but plenty of people have never used protos. They would shrug proto away if there weren't someone like me to attest that it won't get in the way. It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware.

[bufbuild]

> No clear and easy way to use it with HTTP/1.1. Neither is specific to Python.

> It seems like Google focused on gRPC, but the real prize for adoption would've been going after the simple HTTP+JSON use cases. Like an official protobuf Express middleware.

We'd flag https://connectrpc.com for you :-) HTTP/1.1 out of the box, switches on Content-Type between binary and JSON, and fully-compatible with gRPC. cURL with JSON just works.

[bigcityslider]

Yeah I saw that a while back, this is how it should be done