[honkhonkpants]

Could you expound upon the problem of keeping your protocol definitions in sync? In my experience this is the strength of protocol buffers: if you follow a few rules, your systems can successfully be decoupled. Some of the rules are never re-using a tag number and never changing a type in an incompatible way (e.g. string->bytes might be ok, but int32->bytes is not).

[manacit]

Yeah! I think a few responses have covered this below, but I'll give you our spin (and why it's painful, compared to what people have offered up).

Most of the projects that we're integrating gRPC into are existing codebases that have their own build tools that are (mostly) in isolation. JSON schemas have been agreed on beforehand, and there are separate client implementations in different languages that basically exists as independent units.

By adding protobufs to this process, the "JSON schemas that have been agreed on" become protobuf definitions - which is _fantastic_ for development teams, because they have a single spec to work from, and there is no ambiguity (or, significantly less).

The challenge comes when we are trying to generate gRPC clients in Go, Ruby and Python for the same profobuf file - in order to do it in an automated fashion without a 'monorepo', we need to create a build system that pulls this protobuf from a central place and generates the client, which doesn't exist right now.

It's not a huge challenge to ensure services can communicate at all - as you said, protobufs have thought of this and have an extensive amount of decoupling built in. When we're working on adding new features however, we need to have a place to keep the "gold master" of protobufs, and grab it for all of our projects to build+deploy at once, which is where the above becomes challenging.

Not an un-solvable problem, and different languages have different tooling for this. We've settled on placing the proto definitions in the "server" side (most of our interactions are fairly well modeled by client/server), and then updating the clients as-necessary, as we can deploy server changes without needing to update the clients immediately.

[GauntletWizard]

There's no need to pull the proto file on every build. Proto also has a set of rules for how to maintain wire-compatibility across versions[1]. Following those rules and distributing the definition only when you need new fields should be sufficient.

That said; If you've got a set of shared proto definitions, you should probably either go to a monorepo, or share the shared bits with a git submodule. Doesn't prevent you from needing to follow those conventions, but does make it far easier to debug when things changed.

[1] https://developers.google.com/protocol-buffers/docs/proto3#u...

[teen]

He's got the same protos checked into multiple projects under different files. They need to be kept in sync

[jcrites]

That problem can be avoided without monorepos. You primarily need a way to declare a dependency from one package on another at build time, such that the appropriate release gets pulled in. For example, maybe you've depended on version 2 of the interface definition; and in that case the build system fetches the artifacts for interface 2 at build time when building the client. Maven for Java works this way.

Ideally this system would also allow the package owner to release updates within an existing version if they wish. For example, backward-compatible changes to the service interface can be released while keeping the major version 2. In this way, clients automatically consume safe updates, while incompatible or risky changes can be given a major version bump (e.g. to 3). Consumers who want to pin the interface to a specific version like 2.5.1 could do so, in some build systems, though dependencies this specific are rarely useful or a good idea. In my experience it's best for the contract between producer and consumer to be explicitly versioned at the "major version" level, and only implicitly versioned (meaning updates are automatic) at the minor version level.

[honkhonkpants]

That seems like a "doctor it hurts when ... " scenario, and I don't see why it's specific to protobuf. Any IDL managed that way would have the same problem.