[rektide]

I wonder if the original protobuf authors realized what they were doing would be carved in stone & become by far the leading structured data interchange system. If they had any idea so many subsystems would grow up around it, that it would root an ever-expanding ecosystem of tools?

[kentonv]

Certainly not. The very first versions of Protobuf (years before I got involved) were obviously just a quick hack thrown together to get a job done. Everything that seems intrinsic to Protobufs today (or even in 2005) had actually been built up incrementally over several years. E.g., the first versions did not involve code generation. Instead, there was a class that would build messages with the caller manually specifying tag numbers and associated values. This class was called "ProtocolBuffer", which actually made sense as a name for that class. The "buffer" part of the name has long since lost its meaning when talking about the overall product.

The creators (Jeff and Sanjay, two of the most-senior and most-prolific engineers at Google) were fully aware that some of the choices weren't the best, e.g. "varint" format for variable-width integer encoding is simple but not particularly well-designed from a performance perspective. They knew this, but it wasn't worth optimizing at that moment. Now it's way too late to change.

But that's OK. Systems that evolved often have such warts, yet function much better overall than systems which are carefully designed, because systems which evolved will naturally prioritize the design decisions that are actually important to get stuff done. If you sit down and try to design the perfect system in isolation, no matter how smart you are, you will inevitably spend a lot of effort on some features that don't matter, and not spend enough time on some that do. Protobuf is not an ideal design but at this point it has had so much effort put into making it practical that it's hard for anything else to catch up.

(Disclosures: I wrote Protobuf v2 and did the first open source release. I also wrote Cap'n Proto, a "better" design which turns out to be less practical in most use cases because it lacks the ecosystem breadth of Protobuf. I am also an early, small investor in buf.)

[rektide]

Kenton, apologies for the OT question/prod, and I dont remember where but I heard you saying there just hasn't been justification/demand for it- a good use case yet- but still really hoping to see being able to pass references between systems, expand more into three+-party rpc.

Do you have any thoughts on whether QUIC / HTTP3 might be cause for shakeup? Surely grpc can be made to take advantage of http3, but that feels incremental. Do you feel like you would made made captnproto the same way you did if QUIC was an established technology when you began?