[summerlight]

https://news.ycombinator.com/item?id=18190005

Just FYI: an obligatory comment from the protobuf v2 designer.

Yeah, protobuf has lots of design mistakes but this article is written by someone who does not understand the problem space. Most of the complexity of serialization comes from implementation compatibility between different timepoints. This significantly limits design space.

[thethimble]

Relatedly, most of the author's concerns are solved by wrapping things in a message.

> oneof fields can’t be repeated.

Wrap oneof field in message which can be repeated

> map fields cannot be repeated.

Wrap in message which can contain repeated fields

> map values cannot be other maps.

Wrap map in message which can be a value

Perhaps this is slightly inconvenient/un-ergonomic, but the author is positioning these things as "protos fundamentally can't do this".

[evanmoran]

To clarify. Protobuf’s simplest change is adding a field to a message so wrapping maps of maps, maps of fields, oneof fields into a message makes these play to its strengths. It feels like over engineering to turn your Inventory map of items into a Inventory message, but you will be grateful for it when you need a capacity field later.

[missinglugnut]

>Most of the complexity of serialization comes from implementation compatibility between different timepoints.

The author talks about compatibility a fair bit, specifically the importance of distinguishing a field that wasn't set from one that was intentionally set to a default, and how protobuffs punted on this.

What do you think they don't understand?

[summerlight]

If you see some statements like below on the serialization topic:

> Make all fields in a message required. This makes messages product types.

> One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Surely that's a win, isn't it?

> Granted, on paper it's a cool feature. But I've never once seen an application that will actually preserve that property.

Then it is fair to raise eyebrows on the author's expertise. And please don't ask if I'm attached to protobuf; I can roast the protocol buffer on its wrong designs for hours. It is just that the author makes series of wrong claims presumably due to their bias toward principled type systems and inexperience of working on large scale systems.

[instig007]

> If you see some statements like below on the serialization topic:

> Make all fields in a message required. This makes messages product types.

> Then it is fair to raise eyebrows on the author's expertise.

It's fair to raise eyebrows on your expertise, since required fields don't contribute to b/w incompatibility at all, as every real-world protocol has a mandatory required version number that's tied to a direct parsing strategy with strictly defined algebra, both for shrinking (removing data fragments) and growing (introducing data fragments) payloads. Zero-values and optionality in protobuf is one version of that algebra, it's the most inferior one, subject to lossy protocol upgrades, and is the easiest one for amateurs to design. Then, there's next lavel when the protocol upgrade is defined in terms of bijective functions and other elements of symmetric groups that can tell you whether the newly announced data change can be carried forward (new required field) or dropped (removed field) as long as both the sending and receiving ends are able to derive new compound structures from previously defined pervasive types (the things the protobuf says are oneoffs and messages, for example).

[xyzzyz]

What you describe using many completely unnecessary mathematical terms is not only not found in “every real-world protocol”, but in fact is something virtually absent from overwhelming majority of actually used protocols, with a notable exception of the kind of protocol that gets a four digit numbered RFC document that describes it. Believe it or not, but in the software industry, nobody is defining a new “version number” with “strictly defined algebra” when they want to add a new field to an communication protocol between two internal backend services.

[instig007]

> What you describe using many completely unnecessary mathematical terms

Unnecessary for you, surely.

> Believe it or not, but in the software industry, nobody is defining a new “version number” with “strictly defined algebra” when they want to add a new field to an communication protocol between two internal backend services.

Name a protocol that doesn't have a respective version number, or without the defined algebra in terms of the associated spec clarifications that accompany the new version. The word "strictly" in "strictly defined algebra" has to do with the fact that you cannot evolve a protocol without strictly publishing the changed spec, that is you're strictly obliged to publish a spec, even the loosely defined one, with lots of omissions and zero-values. That's the inferior algebra for protobuf, but you can think it is unnecessary and doesn't exist.

[tptacek]

Instead of just handwaving about whether it's necessary or not, why not point to any protocol that relies on that attribute, and we can then evaluate how important that protocol is?

[porridgeraisin]

Yeah. And for anyone curious about the actual content hidden under the jargon-kludge-FP-nerd parent comment, here's my attempt at deciphering it.

They seem to be saying that you have to publish code that can change a type from schema A to schema B... And back, whenever you make a schema B. This is the "algebra". The "and back" part makes it bijective. You do this at the level of your core primitive types so that it's reused everywhere. This is what they meant by "pervasive" and it ties into the whole symmetric groups thing.

Finally, it seems like when you're making a lossy change, where a bijection isn't possible, they want you to make it incompatible. i.e, if you replaced address with city, then you cannot decode the message in code that expects address.

[summerlight]

> since required fields don't contribute to b/w incompatibility at all, as every real-world protocol has a mandatory required version number that's tied to a direct parsing strategy with strictly defined algebra

At least I know 10 different tech companies with billion dollars revenue which does not suit to your description. This comment makes me wonder if you have any experience of working on real world distributed systems. Oh and I'm pretty sure you did not read Kenton's comment; he already precisely addressed your point:

> This is especially true when it comes to protocols, because in a distributed system, you cannot update both sides of a protocol simultaneously. I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths, leading to ugly and hard-to-test code. Inevitably, developers are pushed towards hacks in order to avoid protocol changes, which makes things worse.

I recommend you to do your homework before making such a strong argument. Reading a 5 mins long comment is not that hard. You can avoid lots of shame by doing so.

[b_e_n_t_o_n]

Is this satire?