[louthy]

I find this to be a really poor argument. Essentially you're lucky if your systems continue to work as others go off changing message formats without consideration for the code that will receive it?

On a suitably complex/large system this is a recipe for disaster. Things start to slowly rot. It is far better to maintain the old function, accepting the old struct, map it to the new struct and forward it on to the new function that accepts the new struct. Let the old one consume anything that's already queued, or being sent from other nodes that haven't yet been upgraded whilst the new one takes the new format.

[cousin_it]

I've worked with systems like that for years, and it's fine. We can have several large binaries with different release schedules, passing around a big struct with 50 fields and many nested structs, with different people making changes to different parts. And nothing breaks. New code accepts old structs, old code accepts new structs, no conversion code required.

To achieve that, we follow the design of Protocol Buffers:

1) Each field in each struct has both a name and a numeric id. Only ids are used for serialization, so field names can be changed at any time.

2) All fields are marked as optional or repeated, never required. Most code is written to handle missing fields gracefully.

3) Changing the type or id of an existing field is forbidden. (Note that changing the contents of a nested struct doesn't count as changing its type.)

4) Adding a new field is okay, as long as you use an id that was never used before. (Each struct definition has a comment indicating the next available id to use.)

5) Removing a field is okay if you've checked that no one is using it anymore.

6) As a small but intentional bonus, you can change an optional field to repeated while preserving binary compatibility.

In the end it works out. You can think of breakages that could theoretically happen, but they don't.

[louthy]

What you're describing sounds like a manually implemented type system.

> Each field in each struct has both a name and a numeric id. Only ids are used for serialization, so field names can be changed at any time.

Fair enough your field names can be renamed. But the 'contract' is field numbers, not names.

> All fields are marked as optional or repeated, never required. Most code is written to handle missing fields gracefully.

So if all fields are optional, and you provide no fields at all, what happens? I assume the process rejects it, because it's not of the correct type?

> Changing the type or id of an existing field is forbidden.

Forbidden by what?

> Adding a new field is okay, as long as you use an id that was never used before. (Each struct definition has a comment indicating the next available id to use.)

I can understand this being the least problematic change to a type. But it still leads to 'if x has y field' behaviour, as your code tries to manage the full range of possible message types it might receive.

> Removing a field is okay if you've checked that no one is using it anymore.

That sounds super fluffy.

> As a small but intentional bonus, you can change an optional field to repeated while preserving binary compatibility.

Sorry, I don't follow? This bit confuses me 'change an optional field to repeated'.

> In the end it works out. You can think of breakages that could theoretically happen, but they don't.

I can think of many:

* If picking of IDs is done by a human, at some point a human will make a mistake and re-use an existing one

* If 'Changing the type or id of an existing field is forbidden' is a human enforced constraint, then it will fail

* If you think a certain struct pattern can't happen any more (you think you've retired all nodes that send the old format), and then you deprecate the many matches that deal with legacy messaging, and then realise that actually there is an old node that does it after all.

* You may re-add a field to a type which was previously removed and cause unexpected behaviour in parts of the system that match on that old format

* Removing a field that you thought wasn't used any more but actually still is

By the way, I'm not suggesting it's not possible to develop robust systems without a static type system of some sort; but I do think the hoops you're jumping through in items 1-6 indicate the problems of not using static types. Each change in functionality could just use a new struct, with a new function, and the old function maps to the old struct to the new one. It captures precisely the change in logic in one place, has no runtime cost for nodes that are sending the new struct, and can't lead to the edge cases that I listed above.

[cousin_it]

It helps that we have one big source control repository for the whole company, which runs many custom hooks that can block changes from being committed. For example, if I try to add a field whose id is lower than the next available id specified in the .proto file, my change won't commit. Same if I try to remove a field that's still used by someone else and my change breaks their build.

More generally, I think RPC interfaces need to be forward-compatible by design. If you have two binaries that are released on different schedules, and the API between them is fully rigid, how do you ever change it? Version the whole API, for a change that adds one boolean feature flag to one struct somewhere? Write a converter for fifty existing fields every time you add a new field, leading to O(n^2) programmer work over time? Come on.

Even more generally, I think static types are a great idea, but they work best locally. Communication over longer distances (in space and time) requires a different set of tradeoffs. There's a reason why people design network protocols and file formats with open-ended forward compatibility in mind. RPCs are kind of a middle ground, and I've found that the tradeoffs in Protocol Buffers work pretty well. YMMV.