Hacker News new | ask | show | jobs
Designing Network Protocols (journal.paul.querna.org)
66 points by tomazmuraus 5236 days ago
5 comments

As someone who troubleshoots networks for living, it can't be overstated what value easily understood information has. Normal factors such as lack of understanding of a system, miscommunication, false assumptions, false information, all types of bugs, and operator error already plague troubleshooting. Anything you can do to simplify is extremely important. Little things like being able to read debugs directly out of wireshark can make the difference between solving something between minutes and hours, days and weeks.

Unfortunately serviceability is not normally high in the initial product requirements, but I've seen a direct correlation between customer satisfaction in support and products/protocols/designs with good serviceability.

I think the short version is: in the past 5 years, simple wire-oriented serialisation formats have become much more common. At the time you had your pick of ASN (humongous) or Thrift (brand new).
Not particularly fair. You also had XDR (ubiquitous on Unix systems), IIOP, TLV, ICE... not to mention what ever protocol designer for the past 20 years has used: network byte order integers and ASCII/UTF8 strings.

Some people just like ASCII, human readable protocols. There's nothing wrong with that, but it's a little silly to suggest that the options for a packed binary encoding in 2007 were limited because Thrift and Protocol Buffers were too new.

You're right, to my shame. I should've thought of XDR myself, after suffering through a lecture on it at uni.

I'd be interested in the original designer's remarks on using existing wire serialisation formats.

I mentioned ASN.1 DER, but I honestly didn't think I should go into a history of XDR or other encodings. I guess I can't skip any history in a blog post....
I was responding to a comment, not your post. I don't think you really need to justify using an ASCII protocol (though, again, I think HTTP query arguments are a poor choice).
I don't believe a tl;dr is necessary for this short article.

I enjoyed reading through his thought process for designing a simple protocol which is:

  * Easy to use (requiring little/no additional libraries)
  * Easy to extend (simple keyword/value extensions)
  * Immune to changes in technology
  * (above all) easy to understand
Yet he himself says he considered and rejected a binary protocol for the reason I gave: "I considered using a binary format, but the immediate problem was having extendable fields.", going on to point out that he rejected Thrift because it was too new and ASN.DER because it was too big.

That said, I think he didn't want a binary format in any case -- his "doing it again today" remarks point to JSON.

Actually, the short version is: I wanted it easily debuggable by a network admin. (It's a point he made repeatedly). All the other arguments were moot.
So the network protocol must be in text so that a network admin could debug it? This is absurd.
How does a network admin debug a binary protocol for which no dissector has been implemented/merged into core for Wireshark, and no decoder has been written for tcpdump?

It's obviously doable, but it's very painful.

Isn't Wireshark extensible in Lua?

I can see both sides of the argument here, but basing a protocol on text just for the ease of eyeballing it on-the-wire seems like optimizing for the uncommon case.

Heck, almost any decent protocol should only have ciphertext on-the-wire anyway.

That's more or less like saying "well they can just write the decode". They're network administrators. If you use an ASCII protocol, they don't have to do anything.
How does a sysadmin debug a binary application for which he doesn't have any symbols?
On the other hand, are Wireshark and tcpdump now the gatekeepers for new protocols?
What's your point? I'm not making a value judgement.
Not as absurd as you'd think. You don't want to debug the protocol itself, but you want to be able to easily read what messages were exchanged.

I get the rationale. But I think it's weak, and this entire post is lots of fluff around that core rationale. (I've been writing extensible binary protocols back in 1988 - and it never struck me as particularly difficult even back then.)

There's a complete chapter in The Art of Unx Programming about the importance of being textual:

  http://catb.org/~esr/writings/taoup/html/textualitychapter.html
HTTP-style query strings are a horrible format, whether you like ASCII or not.
I would accept the ease of debugging argument if these messages weren't so small and so common. 1000 servers constructing strings and sending them over the network once a second is a nontrivial waste of resources.