[9dev]

While JMAP seems to scratch every itch of a sucker for proper web API design, I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP. Is there really any new-ish binary protocol these days? Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO. Then again, a lot of solid thinking went into these things, so there probably are a lot of good reasons that I’m not aware of.

Still, it’s an interesting space, I think.

[WorldMaker]

> binary protocol

Email was never a binary protocol. Notoriously so, it's why MIME types and MIME encodings get so complicated.

Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet. HTTP as the new telnet has a bunch of improvements when it comes to binary data, request/response-based data flows, and some other considerations. HTTP/3 is even inherently a binary protocol, it's lack of "telnet-compatibility" one of the concerns about switching the majority of the web to it.

vCard/vCal/iCard/iCal were also deeply "plaintext formats". JSON is an improvement because it is more structured, even more efficient, than those predecessors. JSON may not look efficient, but it compresses extremely well and can be quite efficient in gzip and Brotli streams.

I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet", even if it doesn't sound like "binary protocol efficiency" at first glance. Especially as HTTP/3 pushes HTTP more efficient and more "binary", and arguably "more fundamental/basic" with HTTP/3 even taking over more roles in the TCP/UDP layer of the internet stack. (Telnet would never try to replace TCP.) HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of. Sure, it would be neat to see more variety and experiments outside of the HTTP stack, too, but HTTP is too useful at this point not to build a bunch of things on top of it instead of as their own from-scratch protocol.

[p_l]

A lot of the textual nature of older IETF protocols, including the CR LF line endigns, can be probably traced to how easy it was to bang out a bad implementation full of subtle problems that could be debugged by sitting an undergrad student at a teletype instead of spending time on having some binary serializer (that telecom companies definitely had money for)

[JoshTriplett]

Yeah, a fair bit of email protocol reeks of "is this tolerant of `telnet mailserver 25` and whatever garbage that might produce".

[p_l]

A lot of old RFCs explicitly mention running on top of TELNET.

Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and one of the earliest ways to access ARPAnet outside of machines directly on it was through calling into a TIP which set up bidirectional stream from your modem to a port on some host.

[tbrownaw]

> Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and

The idea with packets is that you don't need to reserve N bit/s of each link along the route to whatever system you're talking to; instead you just repeatedly say "here's a chunk of data, send it to X". It's not really relevant that the typical thing to do with these packets is to build a reliable stream on top of them, what matters is that everything except the endpoints can be a lot dumber.

[miki123211]

There's something in-between traditional (IP style) packets and traditional (TDM / ISDN style) circuit switching, where you use packets with no bandwidth guarantees, but address them by connection number, not endpoint address.

This still requires you to set up a connection beforehand, but doesn't require you to reserve resources you might not be using.

[p_l]

And while certain telcos had packet network focused on that (Bell and Datakit, for example), and generally OSI cared about including QoS indicators, CLNS and its specific on-the-wire implementation CLNP is so similar to IP that IPv9 explicitly called it out in the proposal to replace IPv4 with it (because of ISO 160bit addressing)

[cyberax]

It's also a reflection of the state-of-the-art at the time. Binary protocols were an unmitigated disaster, with standards bodies thinking that applying to the ISO for your organizational ID is a perfectly fine step that anyone does anyway.

[zzo38computer]

> with standards bodies thinking that applying to the ISO for your organizational ID is a perfectly fine step that anyone does anyway

Fortunately there is the 2.25 OID arc now, which you can use without any registration with anyone. There are also other ways to register OIDs for free. (I think that it is better than using domain names, which can be reassigned, and also require registration anyways. IDN is an even more severe problem (it could have been designed better, but they made it worse instead).)

I had idea (which would have to later be made standardized by ITU or ISO (preferably ITU)) of a new OID arc which allows you to combine an existing identifier (of many different types, such as: international telephone numbers, amateur radio call signs, internet domain names (encoded as bijective base 37), IP addresses, ICAO airport codes, etc) with a timestamp, and optional auto-delegation. (You can then add additional numbers like you can with other OIDs too)

Binary protocols have other benefits as well, such as not requiring escaping, and allowing binary data to be transferred is a way that is not as messy, not causing problems with character sets, etc.

[p_l]

That's why there was an entire section for unassigned numbers.

Binary protocols just meant you actually needed to implement serialiser/deserialiser and similar tooling instead of writing dumbest possible riff on strtok() and hoping your software won't be used anymore once DoD internet becomes mature

[cyberax]

> That's why there was an entire section for unassigned numbers.

That's also why the majority of OIDs in SNMP are rooted in the 1.3.6 hierarchy, which belongs to the DoD.

[p_l]

SNMP OIDs are in 1.3.6 because TCP/IP stack was once more well known as "DoD Internet".

And SNMP is explicitly a DoD Internet simplified alternative to CMIS

[p_l]

Another point is that the use of HTTP for everything, outside of the issue of middle boxes breaking protocols for everyone, is that it's essentially capitulation to the wisdom of OSI multi-layered protocols - we replicate their feature sets by reusing bits and pieces of HTTP spec all the time.

[9dev]

It’s not that I cannot appreciate the improvements in the space, I’m just wondering if there might be a big part of the design space for widely used protocols that ends up unexplored because the default for almost anything now is HTTP. It has basically become OSI layer 8 at this point.

[p_l]

MIME/Content-Negotiation is essentially OSI Presentation Layer

HTTP sorta acts as stump of ROSE with bit of ACSE. In addition it provides a bit of basic layer for passing some extra attributes that might be considered in-band or out (or side?) band to the actual exchange.

[WorldMaker]

HTTP/3 replaces parts of OSI layer 4 (TCP/UDP). There's a perspective that HTTP has been trying to lower its layer for decades. Especially the way that TLS originated as HTTPS but now is almost a "universal" vertical chunk of Layers 4, 5, and 6. In that perspective HTTP is probably the new Layer 6.

[jeroenhd]

HTTP/3 uses QUIC as a layer 4 equivalent (which is actually just UDP with extra implications in practice). Not many services leverage standards-compliant QUIC other than HTTP/3, but you don't need to do HTTP to get the same protocol working.

If anything, HTTP/3 running on top of QUIC forced shitty middlebox vendors to de-ossify by permitting any QUIC-based protocol, as they cannot practically distinguish a new HTTP/3 connection from a QUIC connection.

[afiori]

My understanding is that they cannot even really recognise a QUIC connection (maybe the first time, but not after ip roaming)

[thequux]

Ehh, not really. OSI layer 5 was responsible for managing multiple non-overlapping sessions within a single transport stream, and routing those sessions to specific applications. This is precisely what HTTP/1.1 did (though the accept, content-type, accept-encoding, and transfer-encoding headers are really an implementation of layer 6); QUIC, on the other hand, covers the same layers that TCP does (3-5) plus the aforementioned layer 6.

I recommend actually reading X.200 (the specification of the OSI model) at some point: it's quite approachable (especially for an ITU spec, which are notoriously dense reading), and will quickly make you realize how silly it is that we still use it as a reference for modern stacks.

[zzo38computer]

> Notoriously so, it's why MIME types and MIME encodings get so complicated.

I made up ULFI because I thought MIME has some problems.

> JSON may not look efficient

Efficiency is not the only issue; there is also the consideration of e.g. what data types you want to use. JSON does not have a proper integer type, does not have a proper binary data type (you must encode it as hex or base64 instead), and is limited about what character sets can be used.

(Also, like other text formats, escaping will be needed.)

> I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet"

I think it can be, depending on the specific use; sometimes it isn't, and will make it worse. (HTTP does have the advantage of having URLs and virtual hosting, although I think it adds too much complexity more than should be needed.) However, I still think that DER is generally better than JSON.

> HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of.

I think it depends on the specific application. However, even then, I think there are better ways than using HTTP with the complexity that it involves, most of which should not be necessary (even though a few parts are helpful, such as virtual hosting).

[dotancohen]

  > JSON does not have a proper integer type

What are the drawbacks to using the JavaScript Number (really a double float I think) datatype as an integer in an object representation language such as JSON? I've never seen a use case where e.g. 42 (int) could be confused with 42.0 (float). If your application needs specifically an int or a float, then the ingesting application knows that.

If the answer is monetary values, then those should never be floats, and should not be represented in JSON as such. E.g. a dollar and a half should be represented as 150 cents. This follows even for sub-cent precision.

[cstrahan]

JSON numbers are not JavaScript Numbers.

While the grammar is specified (that’s what JSON is, after all), the runtime representation is unspecified. A conformant JSON parser can parse “1” as 1.0. They can be backed by doubles, or singles, or arbitrary precision.

[dotancohen]

That's exactly my point. If one needs an integer or a float, the parser converts it. I ask under what use case is there ever ambiguity?

[cstrahan]

> If one needs an integer or a float, the parser converts it.

Which parser? That’s the problem: if you’re using JSON as a data interchange format, you’ll need to carefully control both the serializers and deserializers, and whatever libraries you use, they will need to (at least internally) hold onto the number in a lossless way — I am not aware of any libraries that do this. They all parse the number as an f64 before any deserializers run. If your input JSON contains a u128, then you’ll have a loss of precision when your type is deserialized.

If you can set up (de)serialization to work the way you need it, then there’s no problem. But if you share your JSON serialized data with other parties, then you/they may be in for a bit of a surprise.

You might find it a worth while exercise to try parsing JSON containing an arbitrary unsigned 128 bit integer in your language of choice.

[funcDropShadow]

Even if we assume that JSON numbers are JavaScript numbers. There is the problem that some large natural numbers cannot be represented in double or float although some even larger numbers can be represented. This is very bad if you use these numbers as IDs.

    scala> (Long.MaxValue-1)
    val res4: Long = 9223372036854775806
                                                                                
    scala> (Long.MaxValue-1).toDouble.toLong
    val res5: Long = 9223372036854775807

The fact that I used Scala is irrelevant here. That is true for many programming languages that 64 bit long and double types.

[dotancohen]

But that problem is not inherent to JSON, which is a pain text encoding of those numbers.

[zzo38computer]

I don't mean 32-bit integers, but rather, 64-bit (and sometimes bigger) integers. JSON already works OK for 32-bit integers (although it is not ideal).

[RedShift1]

Much easier would just be native decimal support in JS and we can represent a decimal using d as the decimal point, for example 15d0 (15.0) or 384d25 (384.25).

Using cents instead of dollars sounds fine until you have to do math like VAT, you really need decimal math for that.

[afiori]

Many applications simply use strings for that, like for dates

[frumplestlatz]

HTTP is also a large, complex stack for any server/client that isn’t already a web server or a browser.

[zzo38computer]

Yes, this is the other issue with using HTTP.

[7bit]

> Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet.

That's just not true. Telnet and SMTP are built on top of TCP. They live on the same layer. They were originally both protocols that transmitted data with printable ascii, hence why they look similar. There are many other protocols like Telnet and SMTP that worked like that, auch as nntp, irc, and yes, even http.

[20after4]

I think that the point is this: You could literally use a telnet client to talk to all of those servers and it was considered a good thing, maybe even an essential feature. So protocol design was somehow influenced by the need to be fully plain text ASCII streams (and maybe that other options weren't adequately explored due to this restriction)

[7bit]

Yeah, I got the point. His statements are still factually incorrect and needed correction.

[dhosek]

You can also telnet into port 80 and communicate with an HTTP server.

[WorldMaker]

Unless it is HTTP/3.

[gsich]

or 2.

[zzo38computer]

> I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP

It shouldn't. For some cases it helps, but other times it doesn't. Sometimes it helps but there would be better ways to do it, making it on a simpler protocol or making an entirely new protocol (which might or might not use TCP; sometimes it is better to use TCP and sometimes not) depending on the specific case.

> Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO

I dislike JSON. I think it has many problems, and that DER is a better format.

(There are also the "small web" protocols such as Gemini and Scorpion and Spartan and Titan, which avoids some of the complexity of HTTP; I had considered using DER-over-Scorpion rather than JSON-over-HTTP. It is also possible to use SSH, although SSH does not have virtual hosting.)

[afiori]

Is der the same as ASN.1?

[zzo38computer]

DER is one of the formats of ASN.1 (in my opinion it is the only good one).

[SchemaLoad]

How much overhead do you think fetching emails over http takes? Fetching text documents with HTTP seems like the prime usecase for it. And it's the only way you could have something that works in web clients.

I'm struggling to think of any real benefits to not using HTTP other than it would be more interesting.

[toomim]

> Is there really any new-ish binary protocol these days?

HTTP/2 and HTTP/3 are binary protocols. And if you replace the JSON with CBOR, then even the payload becomes binary.

The reason for using HTTP is that the semantics are right. HTTP is a state transfer protocol, and ultimately, that's 90% of what you need for sync.

The other 10% is for subscriptions, updates, with versioning, and patches. You can get these by adding the Braid extensions (see braid.org) which upgrade HTTP from a state transfer to a state synchronization protocol. (I work on Braid.)

[solarkraft]

> I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP.

Absolutely yes, IMO. This significantly eases web client development.

[lxgr]

Not being tied to well-known port numbers, and by extension a single service per IP address, is also great.

[solarkraft]

You can run HTTPS on any port.

[lxgr]

My argument was in favor of layering protocols on top of HTTP(S), which allows hosting an unlimited number of instances of servers for the same protocol on the same IP without having to somehow agree on using a non-standard port.

[pjc50]

People think serializers are easy, when suddenly this makes a whole raft of choices your problem. You can write your own, and now you have two problems; or you can build on top of capnproto or grpc or, if you really want to get retro, ASN.1 and inherit the problems of your serializer framework.

JSON by comparison has simple, obvious limitations that more people are familiar with dealing with.

There's also the tendency to tie your protocol to implementation. The Microsoft Exchange "protocol" didn't get reverse engineered for so long because it's basically the COM structure of Outlook fed through (if I remember rightly) DCOM-RPC.

[wmf]

I wonder if you could transparently upgrade to CBOR over HTTP/2.

[btown]

My hot take is that if the web had been based on binary protocols rather than HTML and JSON, the fragmentation of tooling that can instead be standardized as view-source and network-tabs would have decimated (as in removed 90% of) the hackers that make up this community, and software engineering in general.

There's no magic. Nothing sacred. Nothing that you aren't allowed to understand, intuitively. Nothing where you aren't allowed to imagine "what if it also had X?" The web is yours. The computer is yours. As an industry, we burn some incremental percentage of bandwidth to give you the keys to the kingdom, and to allow you, new developer, to be one of us.

In an age when LLMs feel like magic boxes to tech-minded people new to development, we need this more than ever.

[brongondwana]

JMAP is only JSON over HTTP because that's what all the libraries support. Any data format which provides hashes and arrays would work fine, over any transport. So you could use CBOR or protocol buffers or whatever, over any channel you like.