[dweis]

How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?

[docker_up]

No, you create conversion routines that convert between different versions of structs. This keeps things well understood with no ambiguity. This is very easy, we were doing this 20 years ago and autogenerating the conversions using ANTLR that parsed the XDR files for ONC/RPC.

[joshuamorton]

This doesn't make sense with the concept of rollbacks.

If I rollout server version 2 and client version 2 which each use a new required field, and then realize that there is some terrible error in server version 2, I can't roll it back to version 1, since it will reject all client calls from v2 clients.

The only way to make it work is to add a translation layer, as you suggest, on the server, wait a while, push the new 'required' client, wait a while longer, and then push the server without the translation layer. That's the "strict ordering with appropriate soak time" GP mentions.

[docker_up]

Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then.

Or you could have client version 2 know how to automatically convert to server version 1, because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1.

You can't do this with protobufs because there is no such concept, you just add optional fields, and ignore them with different versions, and it's chaos.

[joshuamorton]

>Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then.

This depends on the update. If indeed the field was optional, you won't. A common example would be a field that is necessary for a new feature, but without which everything functions just fine, or functions with a minor degredation in experience.

But more importantly, you want to decouple features from api versions and communication protocols. It should be possible to enable an feature without rolling out a new version (for example, via a flipped flag). So a degraded communication protocol shouldn't actually impact anything. The application worked fine an hour ago, adding a new field won't break it.

>You have to roll back the clients as well then.

And if the clients are phones?

>Or you could have client version 2 know how to automatically convert to server version 1

This requires communicating with the server beforehand. Why should I have to roundtrip to negotiate which api version I should use? And what happens if you want to modify the protocol that you use to decide on api versions? Its turtles all the way down.

>because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1.

So now, before I can roll out a server update, I need to roll out a client update that can make sure to negotiate back to a degraded experience until I update the server. Then I have to wait until that new client is rollback safe. Then I can update the server, then eventually I can update the client to remove the shim code. That's the "strict ordering with appropriate soak time" issue that you're still running into.

This has cascading consequences, each server/client update dance has to be mostly atomic, so you can only really do one of these dances at a time, and all clients have to be in sync. If a deep dependency service wants to make an api change, it has to wait until all of the clients are prepared before updating, and if any of those clients is the server in another context, they have to wait until everyone is ready.

That's chaos. And I don't want to do that when the other option is "update the server whenever, as clients upgrade, they'll see the improved service". You avoid the dance. It moves the initiative of an upgrade from the client to the server, and this is a good thing, because there are more clients than servers.

[DougBTX]

> each server/client update dance has to be mostly atomic

There’s a nice write up showing how that’s a risk but not an absolute restriction of using messages with required fields here: https://martin.kleppmann.com/2012/12/05/schema-evolution-in-...

The basic idea is to acknowledge that all systems with forward and backward compatibility will have a translation layer, the question is just how is that defined and implemented?

If all fields are optional, it means that all readers need to handle any field being missing, in other words all readers must be able to process empty messages. A user update message might be missing a user id, and the reader will have to handle that. A couple of options come to mind: do nothing if there is no user id, or return an invalid message error. The key is that this is a translation layer that can noop or error before the message reaches the service business logic.

Then another thought is that message schemes needn’t be defined with version ids, trying to define a strict ordering between message versions is hard as you say, especially when handling non-linear updates, eg rollbacks or readers and writers skipping versions.

Instead, let’s define message schema compatibility. The user message processor could be defined to say it will only process messages with user ids - which practically speaking will be the case regardless of the message definition format - then a message without a user id can be rejected by common message parsing code, without per-service per-field translation code.

With a clear set of compatibility rules, it is even possible to write sensible reusable schema compatibility checking, eg: https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/...

[docker_up]

We agree to disagree. I don't think you can convince me that all optional is better than all required and vice versa, which is okay. My point is required fields makes software age better over the long run because everything is explicit. If you don't agree, that's your prerogative. Everyone thought NOSQL without schemas was a godsend, until their code/service iterated a dozen times, developers leave, documentation gets out of date, and now all their older data can't be read because it doesn't match the code. Same thing holds true for RPC, in my opinion. Yours may differ.

[joshuamorton]

Right, and my point is that all required fields prevents you from iterating. Your software doesn't age at all.

I've never found the problems you describe, and I work with some of the oldest protos around!