[crabbone]

Hey. I wrote another Python implementation of Protobuf. (protopy https://gitlab.com/doodles-archive/protopy it was a while ago and haven't touched it since). I'm not saying it's better than whatever this is or that it's any good, I just post it as a proof of sorts that I'm familiar with the problem.

So, without further ado: Protobuf isn't a standard. You can't have a non-standard implementation of something that doesn't have a standard to begin with. In reality, you have Google's implementation for C++ and then everything else. Everything else was, for the most part, not written by Google. And it doesn't always align 100% with the C++ Google's stuff.

Furthermore, C++ implementation has a lot of idiosyncrasies specific to that language that can't be translated one-to-one into other languages, or, in some cases, shouldn't be, even if they could (eg. C++ implementation is all about source code generation because generating runtime entities s.a. classes in C++ is very difficult, while in languages like Python, generating classes at runtime is easy.)

Furthermore, C++ implementation has a specific way of parsing the binary payload (lazy: only the top definitions are parsed, the inner structure of messages is parsed on-demand). But, is this how every parser should behave? What if you want a SAX-like parser?

----

In the hindsight, I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage. We, as in the whole programming world, don't have good formats in general, and whenever we come to the point of having to use some, we either go with an existing popular but crooked or roll our own, probably also crooked. The standard you alluded to would've been great (perhaps a refinement of ASN with more attention to parser implementation, more concrete versions etc.?) But we aren't there yet, and there isn't even a work group to try and address the issue.

[StilesCrisis]

> I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage.

I am not a fan of Protobuf at all, but it's already demonstrated its ability to ship extremely reliable software with multi-decade lifespans. It's one of the few things Google _hasn't_ deprecated, and it's the backbone of the search and ads stack.

[crabbone]

I don't know what your metric for reliability is... but I would say that the messaging / serialization format can hardly be blamed for reliability problems s.a. service uptime or service responsiveness etc. Such problems usually arise at a level where details like the choice of messaging format are not important.

Messaging format may affect application performance though. It would affect metrics s.a. throughput or bandwidth.

In a way that is more difficult to measure, a messaging format can affect the number of bugs created by developer using it and indirectly the delivery times. But this can be mitigated by tooling (i.e. Protobuf isn't self-documenting, so if you don't have the schema files, you can't interpret the messages, but you can write a tool that you can feed the schema definitions and then use the tool to interpret the messages).

> It's one of the few things Google _hasn't_ deprecated

You might be unaware of it, but there were Protobuf v1, v2, and now we are at v3. Even though it's not documented, v3 supports most of v2, but not all of it (I think v1 was never used outside of Google itself). Google never properly released Protobuf, so, they can't really deprecate it. Even their formal grammar is full of errors.

[fmbb]

Why does it matter for some Python implementation if the Google C++ implementation has a lazy or eager parser?

The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.

Personally I also prefer code generation over dynamic parsers and generators. This is not an idiosyncrasy of C++, it is just the objectively good way to handle IDLs regardless of programming language.

[crabbone]

> Why does it matter for some Python implementation if the Google C++ implementation has a lazy or eager parser?

I wrote about it in my repository, but I'll try to summarize it here: Protobuf is full of bad ideas. One such bad idea is that message fields are allowed to repeat and that the last field wins. So, if you were to write a SAX parser, you'd have a dilemma with how to handle this bizarre idea: do you accept that a callback for some property might be triggered multiple times or do you read the whole message ahead of time and then call callbacks exactly once for each property? If we accept that the parser must be lazy, we "solve" this problem by allowing the parser to read ahead (it needs to do this anyways), but this is a wasteful way to parse (uses more memory than necessary).

> The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.

I'm not sure what are you trying to say here. All messaging formats are made up of... wire format, that's what they are for. Maybe I'm not seeing it?

> Personally I also prefer code generation over dynamic parsers and generators.

I think you are trying to say that you prefer source code generation over generating supporting definitions at runtime? I wasn't talking about dynamic parser generation (eg. Lark). In my case, the parser was hand-written (using Bison + Flex) and compiled ahead of time, but the Python definitions supporting the Protobuf IML were generated at runtime.

If my guess is true, I'd like to hear your arguments in favor of generating Python source code that translates Protobuf IML into Python. To me it looks like a waste of space on disk... I really can't think of any reason to want that.

[squirrellous]

Honest question - why isn’t the following document [1] a standard? Is it too loosely specified?

[1] https://protobuf.dev/programming-guides/encoding/

[crabbone]

It's like in that meme: it's not enough to say it, you need to declare it.

Or, in other words, what makes a standard a standard is that you declare that it is a standard. This declaration carries with it an obligation to respect the standard for ever and ever (like C89), regardless of whether in retrospect you realize you've made mistakes and want to fix them, or maybe wanting to add more stuff etc.

Having a standard is restrictive and uncomfortable for the designers, that's why many opt not to have a standard. Eg. Rust language doesn't have a standard (even though you may, of course, find documents detailing how it works), same for Python, Java and many other popular languages.

[7bit]

Maybe He means because it diesen have an accepted rfc

[usrnm]

At least, Google has the resources and the will to support all their implementations in the long run. I don't always agree with what they do there, but at least I can be sure that it will still work ten years from now. At some point it becomes more important than implementation details

[crabbone]

I don't know where you get the confidence... When it comes to Protobuf, we are now at version 3 of the format. It's been around for a while, but I'm old enough to have implemented v2 parser myself... v2 is partially supported in v3, even though the support isn't documented. But, emphasis on partially. Some things are no longer there.

So... I'd say that your faith is unfounded. And, in general, there's no reason to believe that a commercial entity will commit to supporting any particular technology if that doesn't generate them a profit. Standard is better.

[joshuamorton]

It should be noted that

1. Proto2 isn't actually deprecated or anything and is still widely used and supported

2. https://protobuf.dev/editions/overview/ replace and improve on the versioning concept and basically entirely remove the issue of versioning since versions and features can be incrementally enabled on a per-file and per-field basis.

aaand I guess 3: protobuf is absolutely critical to Google's profit.

[crabbone]

It might be uncalled for, but let me use a philosophical reference: you are trying to derive ought from is. Many tried to prove it to be possible, but most face such propositions with skepticism.

Let me give you some examples of things that were absolutely detrimental to the business of a large company that vanished in a matter of years if not months:

* Adobe Flash. Flash players was for a few years the most installed program on Earth. It vanished without a trace, dropped like a hot potato by everyone who swore they will use it for ever and ever.

* ASP Classic, especially in combination with VBScript. It was the hottest thing twenty (or am I getting too old?) years ago. Everyone was selling books and courses on ASP Classic, probably half the Internet was written in it... and it's gone.

* I expect x86 ISA to eventually die and be completely replaced by ARM or maybe something else. Some big players are already jumping off the bandwagon, and it looks like things are only going to get worse.

There were, of course, more, but I think these three examples should suffice. If tomorrow Google comes up with a better format, or, for whatever reason, tanks its business, or decides to switch to vertical farming in Arizona... say goodby to Protobuf. They owe you zilch. They never promised you anything. You were allowed to use their tech because it was convenient to them, but once it's not, you will be holding the bag.

This is the difference between a standard and a spec published by the designer. The designer is free to make any changes they see fit, up to and including completely destroying their work. They can pull the run from under you making the users pay for the use, they can make modifications to their tools that will require from you to buy things you wouldn't normally want to buy. The possibilities are endless.