[cletus]

Story time: the whole development of protobuf was... a mess. It was developed and used internally at Google long before it was ever open sourced.

Protobuf was designed first and foremost for C++. This makes sense. All of Google's core services are in C++. Yes there's Java (and now Go and to some extent Python). I know. But protobuf was and is a C++-first framework. It's why you have features like arena allocation [1].

Internally there was protobuf v1. I don't know a lot about this because it was mostly gone by the time I started at Google. protobuf v2 was (and, I imagine, still is) the dominant form of.

Now, this isn't to be confused with the API version, which is a completely different thing. You would specify this in BUILD files and it was a complete nightmare because it largely wasn't interchangeable. The big difference is with java_api_version = 1 or 2. Java API v1 was built like the java.util.Date class. Mutable objects with setters and getters. v2 changed this to the builder pattern.

At the time (this may have changed) you couldn't build the artifacts for both API versions and you'd often want to reuse key protobuf definitions that other people owned so you ended up having to use v1 API because some deep protobuf hadn't been migrated (and probably never would be). It got worse because sometimes you'd have one dependency on v1 and another on v2 so you ended up just using bytes fields because that's all you could do. This part was a total mess.

What you know as gRPC was really protobuf v3 and it was designed largely for Cloud (IIRC). It's been some years so again, this may have changed, but there was never any intent to migrate protobuf v2 to v3. There was no clear path to do that. So any protobuf v3 usage in Google was really just for external use.

I explain this because gRPC fails the dogfood test. It's lacking things because Google internally doesn't use it.

So why was this done? I don't know the specifics but I believe it came down to licensing. While protobuf v2 was open sourced the RPC component (internally called "Stubby") never was. I believe it was a licensing issue with some dependency but it's been awhile and honestly I never looked into the specifics. I just remember hearing that it couldn't be done.

So when you read about things like poor JSON support (per this article), it starts to make sense. Google doesn't internally use JSON as a transport format. Protobuf is, first and foremost, a wire format for C++-cetnric APIs (in Stubby). Yes, it was used in storage too (eg Bigtable).

Protobuf in Javascriipt was a particularly horrendous Frankenstein. Obviously Javascript doesn't support binary formats like protobuf. You have to use JSON. And the JSON bridges to protobuf were all uniquely awful for different reasons. My "favorite" was pblite, which used a JSON array indexed by the protobuf tag number. With large protobufs with a lot of optional fields you ended up with messages like:

    [null,null,null,null,...(700 more nulls)...,null,{/*my message*/}]

GWT (for Java) couldn't compile Java API protobufs for various reasons so had to use a variant as well. It was just a mess. All for "consistency" of using the same message format everywhere.

[1]: https://protobuf.dev/reference/cpp/arenas/

[mike_hearn]

There was never any licensing issue, do you think Google would depend on third party software for anything as core as the RPC system? The issue was simply that at the time, there was a culture in which "open sourcing" things was being used as an excuse to rewrite them. The official excuse was that everything depended on everything else, but that wasn't really the case. Open sourcing Stubby could certainly have been done. You just open source the dependencies too, refactor to make some optional if you really need to. But rewriting things is fun, yes? Nobody in management cared enough to push back on this, and at some point it became just the way things were done.

So, protobuf1 which was perfectly serviceable wasn't open sourced, it was rewritten into proto2. In that case migration did happen, and some fundamental improvements were made (e.g. proto1 didn't differentiate between byte arrays and strings), but as you say, migration was extremely tough and many aspects were arguably not improvements at all. Java codebases drastically over-use the builder/immutable object pattern IMO.

And then Stubby wasn't open sourced, it was rewritten as gRPC which is "Stubby inspired" but without the really good parts that made Stubby awesome, IMO. gRPC is a shadow of its parent so no surprise no migration ever happened.

And then Borg wasn't open sourced, it was rewritten as Kubernetes which is "Borg inspired" but without the really good part that make Borg awesome, IMO. Etc.

There's definitely a theme there. I think only Blaze/Bazel is core infrastructure in which the open source version is actually genuinely the same codebase. I guess there must be others, just not coming to mind right now.

Using the same format everywhere was definitely a good idea though. Maybe the JS implementations weren't great, but the consistency of the infrastructure and feature set of Stubby was a huge help to me back in the days when I was an SRE being on-call for a wide range of services. Stubby servers/clients are still the most insanely debuggable and runnable system I ever came across, by far, and my experience is now a decade out of date so goodness knows what it must be like these days. At one point I was able to end a multi-day logs service outage, just using the built-in diagnostics and introspection tools that every Google service came with by default.

[dieortin]

What are those “really good parts” that Stubby and Borg have but their open source versions don’t?

[mike_hearn]

I'll admit it's been a while since I looked at gRPC / Kubernetes and I never used them in anger for real projects. It's possible that some of these claims are wrong or can be filled with plugins, have been fixed in newer releases, etc. Also everything here about Google's stuff is a decade+ out of date. It might all be different now.

One thing that I really miss from other RPC systems is the variety of debug endpoints. Stubby piggybacked on HTTP a bit like gRPC does by registering endpoints into a pre-existing HTTP server. One was a magic hidden endpoint that just converted the socket into a Stubby socket, but others let you do things like:

• Send an RPC by filling out an auto-generated HTML form, so you could also use curl to send RPCs for debugging purposes. There is an OpenAPI based thing that gives you something similar for REST these days, but it's somehow heavier and not quite as clean.

• View all RPCs that were in-flight, including cross-process traces, how often the RPCs had retried etc. This made it very easy to figure out where an RPC had got stuck even if it had crossed several machines. In the open world there's Jaeger and similar, I haven't tried those, but this was built in and didn't require any additional tools.

• View latency histograms of RPCs, connected machines, etc. View the stack traces of all the threads.

• They had a global service discovery system that was basically a form of reactive DNS, i.e. you could subscribe to names and receive push notifications when the job got moved between different underlying machines.

• Endpoints for changing the values of flags/parameters on the fly (there were thousands exposed like this).

• RPC routing was integrated with the global load balancing system.

Probably a dozen more things I forgot.

All this made it very easy to explore and diagnose systems using just a web browser, and you didn't face problems of finding servers that didn't have these features because every app was required to use the in-house stack and all the needed features were enabled by default. Whereas in most open source server stacks the authors are obsessed with plugins, so out of the box they do very little and companies face an uphill battle to ensure everything is consistent.

For clusters the main difference I remember is that Borg had a proper config language instead of the weird mashed up YAML templating thing Kubernetes uses, and the Borg GUI was a lot cleaner and more info-dense than the Material Design thing that Kubernetes had, and the whole reactive naming system was deeply integrated in a natural way. Also Kubernetes is all about Docker containers, which introduces some complexity that Borg didn't have. I had problems in the past with k8s/docker doing dumb things like running out of disk space because containers weren't being purged at the right times, and kernel namespaces have also yielded some surprises. At the time Borg didn't really use namespacing, just chroots.

There are some minor stylistic differences. The old Google internal UI had a simple industrial feel. It was HTML written by systems engineers so everything was very simple, info dense, a few blocks of pastel colors here and there. Imagine the Linux kernel guys making web pages. Meaning: very fast, lightweight, easy to scrape if necessary.

[chubot]

Yeah my memory of this (which is admittedly fuzzy) is that a bunch of business things happened around 2012-2015, which led to these external software libraries / products that are "arguably inferior rewrites" and "not what Google actually uses"

I think 2012 is when Larry became CEO (again), and 2015 is when the "Alphabet" re-org / re-naming happened.

1. Larry Page was generally unhappy with the direction and execution of the company, so he became CEO. (Schmidt would never be CEO again)

2. VP Bill Coughran was shown the door (my interpretation, which is kind of like Eric Schmidt being shown the door). For my entire time there he had managed the software systems -- basically everything in google3, or everything important there

3. Urs Hoezle took over everything in technical infrastructure. I think he had previously been focused on hardware platforms and maybe SRE; now he was in charge of software too.

Urs sorta combined this "rewrite google3" thing with the "cloud" thing. To me there was always a tenuous connection there, at least technically. I can see why it made sense from a business perspective

---

Basically Larry was unhappy with google3 because the company wasn't shipping fast enough, e.g. compared to Facebook. It was perceived as mired in technical debt and processes (which IMO was essentially true, and maybe inevitable given how fast the company had grown for ~8 years)

And I think they were also looking over their shoulders at AWS, which I think by then had become "clearly important".

Why don't we have an AWS thing? At some point GCE was kind of a small project in Seattle, and then it became more important when AWS became big.

Anyone remember when Urs declared that google3 was deprecated and everything was going to be written on top of cloud in 12 to 18 months? (what he said was perhaps open to interpretation -- I think he purposely said something really ambitious to get everyone fired up)

So there was this shift to "externalize" infrastructure, make it a real product. Not just have internal customers, but external ones too.

---

So I think what you said is accurate, and I think that is the business context where the "arguably inferior rewrites" came from

- Kubernetes is worse in many ways than Borg [1]

- gRPC (I haven't used it) is apparently worse in many ways than Stubby, etc.

I'd be interested if anyone has different memories ...

---

[1] although I spent some time reading Borg source code, and e.g. compared to say the D storage server, which was also running on every node, it was in bad shape, and inefficient. There are probably ways that K8s is better, etc.

My main beef is the unimaginable complexity of running K8s on top of GCE on top of Borg -- i.e. 3 control planes stacked on top of each other ...

[cletus]

> do you think Google would depend on third party software for anything as core as the RPC system?

I don't believe Google has (had?) any objections to using open source or open sourcing things but you have to remember two things:

1. Most companies weaponize open source. They use it to "commoditize their product's complements" [1]; and

2. Google3 are so deeply integrated in a way that you can't really separate some of the tech because of the dependencies on other tech. More on that below.

> Open sourcing Stubby could certainly have been done. You just open source the dependencies too

Yeah, I don't think it's always that simple. You may not own the rights to something to be able to open source it. Releasing something may trigger more viral licenses (ie GPL) to force you to open source things you don't want to or can't.

I actually went through the process of trying to import a few open source packages into Google's third party repo and there are a lot of "no nos". Like a project had to have a definite license (that was white listed by legal). Some projects liked to do silly things like having a license like "do whatever you want" or "this is public domain". That's not how public domain works BTW. And if you contacted them, they would refuse to change it even something like an MIT license, which basically emans the same thing, because they didn't understand what they were doing.

> And then Borg wasn't open sourced

This actually makes sense. Later on your suggest you were a Google SRE so you should be aware of this but to whoever else reads this: Google's traffic management was deeply integrated into the entire software stack. Load balancing, DDoS defense, inter-service routing, service deployment onto particular data centers, cells and racks and so on.

It just doesn't make sense to open source Borg without everything from global traffic management down to software network switching.

> I think only Blaze/Bazel is core infrastructure in which the open source version is actually genuinely the same codebase

I don't know the specifics but I believe that Bazel too was "Blaze inspired". I suspect it's still possible to do things in Blaze that you can't do in Bazel even though the days of Blaze BUILD files being Python rather than "Python syntax like" are long gone.

Also, Blaze itself has to integrate with various other systems than Bazel doesn' eg ObjFS, SrcFS, Forge, Perforce/Piper, MPM and various continuous build systems.

[1]: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

[mike_hearn]

Everything that the core stack depended on was written from scratch by Google, there were no third party dependencies with unknown licenses that I ever encountered, or any third party dependencies at all. They started with the STL + POSIX and worked up from there.

I'm pretty sure stuff could have been split out. The architecture was sound and the entanglement was overstated. Nothing would have stopped you bringing up a Borg cluster without the global HTTP routing / DDoS / traffic management stuff, as evidenced by the fact that those parts changed regularly without needing synchronized releases of the other parts.