[withinboredom]

Sounds like Stockholm Syndrome. I've work mostly with JSON/CSV/Thrift in the last 10 years, and xml/soap before that, and just recently started interacting with protobuf, so I'd disagree that it is a "de-facto standard."

My largest complaint: observability. With almost literally any other protocol, if you can mitm on the wire, your human brain can parse it. You can just take a glance at it and see any issues. With grpc/pbuf ... nope. not happening.

Also, I really don't like how it tries to shim data into bitmasks. Going back to debugging two systems talking to each other, I'm not a computer. Needing special tooling just to figure out what two systems are saying to each other to shave a quarter of a packet is barely worth it, if at all, IMHO.

[FridgeSeal]

> You can just take a glance at it and see any issues. With grpc/pbuf ... nope. not happening.

Sure, but on the other hand, the number of times I’ve needed to do this, compared to JSON/string/untyped/etc systems is precisely zero. There’s just a whole lot of failure that are just non-issues with typed setups like protobufs. Protobuf still has plenty of flaws and annoying-google-isms, but not being human readable isn’t one of them IMO.

[hinkley]

I haven’t worked with protobufs, but I’m old enough to know why people thought they needed protobufs (because hand writing terse wire protocols is painfully dumb).

Be careful about “need”. When people are avoiding doing something painful they invent all sorts of rationalizations to try to avoid cognitive dissonance. You don’t reach for to tool that hurts to pick up. You reach for something else, and most do it subconsciously.

Nobody is going to try to read protobuf data. Doesn’t mean they don’t need to understand why the wire protocol fucked up.

[withinboredom]

Nothing is preventing a system from sending you a un-deserializable message disguised as a protobuf, just like with any other encoding. In these cases, you need to diagnose the issue, no? Having something human-readable is simply straightforward.

If you haven't needed to do this, perhaps you aren't working on big enough systems? I've primarily needed to do this when dealing with hundreds of millions of disparate clients, not so much on smaller systems.

[FridgeSeal]

> Nothing is preventing a system from sending you a un-deserializable message disguised as a protobuf,

I guess it depends on where you come down on Postel’s law. If you’re an adherent, and are prepared to be flexible in what you accept, then yeah, you will have extra work on your hands.

Personally, I’m not a fan of Postels law, and I’m camp “send correct data, that deserializes and upholds invariants, or get your request rejected”. I’ve played enough games with systems that weren’t strict enough about their boundaries and it’s just not worth the pain.

[withinboredom]

When you have hundreds of millions of clients, there’s a good chance the client thinks it’s sending you the right data (especially when the customer says everything looks right on their end). You need to figure out if there is packet corruption (misbehaving switch), an outdated client, an egress load balancer screwing with packet contents, an office proxy trying to be smart, etc.

This requires looking at what is going over the wire in a lot of cases. It has nothing to do with Postel’s Law, but more to telling a customer what is wrong and making your own software more resilient in the face of the world.

[hinkley]

Please, please, to anyone following this thread:

Implement a human readable protocol, then use standardized streaming compression on the wire to get your message size down. Something LZ family because there are tools everywhere that speak them. And consider turning off transport encoding for local development.

Being able to scan data saves so much time on triage. And using zgrep and friends on production data is almost as easy. You will spend tons of effort trying to make something 10% more efficient than zlib or for certain zstd, and the cost is externalized onto your team.

[arccy]

human readable == inefficient parse and processing

[hinkley]

It’s IPC. I think are vastly overestimating the local processing overhead.

[arccy]

there's overhead every time you turn a number from native into a human readable decimal representation, and telemetry is mostly numbers.

[Groxx]

This is true with any non-self-describing format. Which includes the vast majority of JSON depending on who you ask - if you aren't specifying a schema in the request, what does `name` really mean?

Self-describing comes with rather large costs over a compact format in essentially all cases, there are lots of good reasons to prefer it. Particularly in internal infrastructure, like telemetry tends to involve.

[secondcoming]

Is compression also bad because you can’t immediately observe the data?

I’ve heard the ‘binary makes debugging difficult’ excuse before and it’s just nonsense really.

[jen20]

Not as bad as encryption if those are the requirements!