[jcgrillo]

I personally really like gRPC and protobufs. I think they strike a good balance between a number of indirectly competing objectives. However I completely agree with your observation that as soon as you move beyond a single source of truth for the .proto files it all goes to shit. I've seen some horrible things--generated code being committed to version control and copied between repos, .proto files duplicated and manually kept up to date (or not). Both had hilarious failure modes. There is no viable synchronization mechanism except to ensure that each .proto file is defined in exactly one place, that each time someone touches a .proto file all the downstream dependencies on that file are updated--everyone who consumes any code generated from that .proto--and that for every such change clients are deployed before servers. Usually these invariants are maintained by meatspace protocols which invariably fail.

[jeffbee]

I don't see why any of that would be necessary. There are simple rules for protobuf compatibility and people only need to follow them. Never re-use a field number to mean something else. Never change the type of a field. That's it. Those are the only rules. If you follow them you don't have to think about any of that stuff that you mentioned.

[jcgrillo]

Absolutely! Forward and backward compatibility are one of the wonderful things about protobufs. And that all goes wrong when you try to define the interface in more than one place.

EDIT: also, although the wire protocol may tolerate unknown or missing data, almost always the application doesn't.

EDIT AGAIN: I'm not saying this is how it should be just that this is the low energy state the socio-technical system seems to arrive at over time. So ideally it should be simple but due to imperfect decisions it gets horribly complicated over time.

[jeffbee]

I fail to see how the application will even be aware of unknown data. Explain what practical problem could possibly arise if you think a message has 4 fields and I send you a fifth one.

Edited to reply to your edits: People who are just bozos with computers will never be kept from bozotry by any interchange format. If they lack any semblance of foresight then maybe they simply should get a different line of work. Postel's law is in force here. If you start sending me emails with extra headers my email program is never going to care. Protobufs are the same way.

[jcgrillo]

Apologies for the delay, this site appears to be rate limiting me. Yeah used correctly they're great. But they're almost never used correctly in practice. I agree this is bozotry in the extreme, but it's widespread. To avoid it all they'd need to do is read like 4 pages of well-written accessible documentation, but sadly that bar is too high. I don't blame protobufs! It's just that, somehow, what should be an elegant, simple system turns into a nightmare in practice. Every. Goddamn. Time. Not unlike when people try to use Kafka. That isn't to say the tool shouldn't be used, just that maybe we need a better way to organize/educate/hire engineers so they don't ruin things so badly. Or at least some way to impose an upper bound the damage they can do. Maybe there's some kind of regularization effect if you force everyone to work with Map<Object, Object> JSON. Or maybe it's just the state everything devolves to eventually.