[IgorPartola]

The obnoxious thing is that overly aggressive firewalls have killed any IP protocols that are not TCP or UDP. Even ICMP is often blocked or partially blocked.

In the mean time we could have had nice things: https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...

SCTP would be a fantastic protocol for HTTP/HTTPS. Pipelining, multi homing, multi streaming, oh my.

[tptacek]

Why would you need another IP protocol besides UDP? Anything you can do directly under an IP header, you can do under a UDP header as well, and the UDP header itself is tiny.

Going back to David Reed, this is specifically why UDP exists: as the extension interface to build more non-TCP transport protocols.

[IgorPartola]

UDP introduced ports. Ports are not always the best abstraction for specifying which application is talking to which other application. They are finite.

I am very aware of what you can do with UDP, I have done some very fun work trying to minimize bandwidth usage on crappy mobile connections by using and abusing it. But I think at the end of the day it is an engineering crutch.

If we insisted on properly supporting a diverse set of L4 protocols years ago we wouldn’t have wound up with NAT and slow adoption of IPv6. Address exhaustion would have been a real pressing issue. Instead you can’t even ping a server behind a NAT and firewalls run out of memory trying to manage stateful connections.

UDP is a pretty elegant design for what it is but it is barely good enough to allow us some room to make things work. Ultimately it did limit us more than it enabled us.

[tptacek]

Ports are just a multiplexing device, the same as the IP protocol number. Besides the tiny number of bytes in the UDP header, what's the practical difference?

[IgorPartola]

Negligible to none as of now. But take a look at this comment below: https://news.ycombinator.com/item?id=45528837

And I agree: it stifled what could have been a much nicer to work with set of protocols and who knows what could have been created had we not just said "well there is always UDP if you want to do your own thing".

[tptacek]

Stifled how? You want SCTP? Allocate a UDP port for it. You still haven't explained what the problem is.

[IgorPartola]

OK let’s put it this way: what is the point of IP being able to carry protocols other than ICMP, TCP, and UDP? For that matter why doN’t TCP and ICMP run on top of UDP?

[jcelerier]

> the UDP header itself is tiny.

we're fighting for bits here!!! if you want to be 100% safe you need 576 packet size, UDP header is 1.4% of that

[supertrope]

IPSec originally ran on raw IP. These days it has to be tunneled in UDP due to TCP or UDP only ossification.

PMTUD breaks when ICMP is blocked.

The same argument can be made that everything but HTTP being blocked is not a problem because everything can be transported on top of HTTP.

[toast0]

It's not nice that pmtud breaks when ICMP isn't available, but sensible probing can do pretty well. Of course, barely anybody probes and they may not be sensible if they do.

Imagine at the beginning of a connection, sending a burst of packets, you could send (ignoring tcp timestamps because it makes the math hard, PAWS is a waste of bytes for most flows, and etc) [0,1460); [1400, 2800), ...

If you get an ack of the first packet, great. If not, you resend it as a 1400 byte payload and probe again in a future burst. Maybe even premptively resend the first packet as a 1400 byte segment after a short delay. Anyway, have enough failed large packets and probe smaller. Probe bigger again every so often if the connection stays open for a meaningful amount of time.

[tptacek]

The same argument is made about HTTP. But at least in the HTTP case, you can point to protocol behavior the middle-layer protocol is enforcing on you. You can't do that with UDP; UDP is just IP, with some ports, and a checksum.

[mtoner23]

Because it's an extra header. Making data transfer that much less efficient and working to make sure that clients can decide it properly

[lxgr]

An extra header of 8 bytes. To put that into perspective, IPv6 added 20 bytes to the IP header compared to v4.

[thadt]

Yeah? It's an eight byte header. The OS needs something to tag IP packets to get them delivered to the correct application. So you're thinking maybe a four byte header for 50% savings here?

[lxgr]

Good point on there needing to be some application-level addressing anyway.

On top of that, I believe the UDP checksum can be omitted as well at least on some OSes (and is arguably not necessary for fully encrypted/authenticated payloads) – leaving really just the two bytes for the "length field".

[tptacek]

You can't omit the UDP checksum in IPv6.

[IgorPartola]

So we have a checksum of the IP header, a checksum of the UDP header and a port number, an application level stream ID or message ID or whatever the application transport protocol is using, and finally almost certainly an even higher level message ID such as a URI. And that’s before you introduce encryption into it with all that overhead. A level 4 protocol providing full integrity verification, encryption, multi homing, multiplexing, out of band control, and control over transmission reliability would be amazing. But the only way you can experiment with these things is if you use UDP and ports. We take the concept of ports for granted but if you think of ICMP or some other L4 protocols that isn’t the only way to identify the sending and receiving application.

If we just allowed all L4 protocol numbers through and ditched NAT we could have nice things. Or we could kick it up two layers to use QUIC for what SCTP could have been.

[tptacek]

There's going to be encryption either way in any modern protocol, and then the header manipulation stuff is already all done in hardware. It's probably more efficient in UDP than as a direct IP protocol, because UDP is fast-pathed in ways protocols other than 6 and 17 aren't.

Having a diversity of IP protocols isn't a nice thing. The designers of TCP/IP made a protocol specifically for doing the thing you wanted to see SCTP do: it was called UDP.

[IgorPartola]

Why isn’t it a nice thing? And SCTP and UDP clearly provide different semantics. I am fine with experimenting with new protocols on top of UDP because it is simple to do but ultimately I think things like SCTP and QUIC should run directly on top of IP.

[yobid20]

^^^^ this. I work for a big company (15k engineers). Trying to use anything that is not TCP or UDP simply doesnt work here. For years, even UDP was blocked and the answer we got was always "why are you using UDP, use TCP instead". Yep you read that right. Most of these folks are very short sighted or narrow minded. We tried to use SCTP for one project, major blunder. Zero support from network teams. Sctp is blocked everywhere. All their custom software and scripts for network deployments only work with tcp and udp. And they will not change that. And that comes from higher ups the ppl in charge. They are set in there ways and will not budge. As for as QUIC support? Never gonna happen here.

[lxgr]

> Trying to use anything that is not TCP or UDP simply doesnt work here.

Good news, then: QUIC uses UDP.

> As for as QUIC support? Never gonna happen here.

So generic UDP is allowed, but QUIC is specifically detected and filtered? That would be bizarre.

[IgorPartola]

I have also had to deal with UDP getting blocked by a middle box before. It is rare now across the internet but you never know what intranet grey beards can do.

And UDP hole punching is a crutch for the fact that we still use IPv4.

[UltraSane]

I'm one of a few people who can say they used SCTP for its original purpose, SIGTRAN transmission of Signaling System 7 call setup data over IPSec tunnels. It was really cool seeing SS7 packets in the wireshark dumps.

[fulafel]

SCTP had some other nails to its coffin too. For example it came from the telco world and so had no end user app buyin and poor dev experience.

[CuriouslyC]

SRT is hot stuff over UDP: https://www.haivision.com/products/srt-secure-reliable-trans...

But even UDP is heavily restricted in most cases.

[lxgr]

Is it really? It's been a while since I've been on a network that actually filters UDP.

Too many things that people actually need to work run over it these days, including VPNs, videoconferencing etc.

[lxgr]

If you like SCPT, nobody's stopping you from using it today! Just put it in a minimal UDP wrapper.

WebRTC has been doing just that for peer to peer data connections (which need TCP-like semantics while traversing NATs via UDP hole punching).

[IgorPartola]

I get today’s realities of it but I would have preferred a world where IPv6 killed NAT and middleboxes properly supported more protocols than TCP and UDP. The original intent of IP was to have more than two protocols built on top of it. Many were built and deployed but then killed by IPv4 address exhaustion and NAT, as well as poorly configured firewalls and middleboxes that specifically wanted to mess with level 4 traffic.

UDP is a good solution but all it does is provide an 8 byte overhead and nothing that IP itself doesn’t provide for something like SCTP.

[lxgr]

IPv6 doesn't provide a length header, so that's already 2 bytes arguably necessary for all protocols layered on top of that.

Source and destination port just seem like a reasonable baseline for alternate protocols, that's 4 more – leaving just the checksum. (If you're really desperate for space and have integrity provided by your protocol, you can even cram two more bytes in there!)

Sure, it would be conceptually nice to be able to skip UDP, but I think in terms of performance it absolutely does not matter.

[IgorPartola]

QUIC doesn’t use the UDP length header to designate message length, does it?

But my point isn’t even about performance. It is about the fact that NAT and IPv4 address exhaustion and bad firewall practices have killed any innovation of level 4 protocols. Imagine if instead of TCP, SCTP had won the protocol wars in 1980-1990s. Or even better if we had realized that we were going to run out of IPv4 addresses much earlier when the cost of switching was smaller. It would have been so much better to have firewalls that don’t filter anything but protocols 6 and 17. We could have had the opportunity to experiment with different types of transports, baked encryption in at a lower level, etc.

Basically where we are is that we have 6 and 8 dot LEGO bricks to play with and are told that we can build anything with those but aren’t allowed to play with any other shapes.

[lxgr]

> QUIC doesn’t use the UDP length header to designate message length, does it?

Does it not? Not sure if it's really mandatory, but I believe one rationale for IPv6 getting rid of both its checksum and length fields was that both TCP and UDP duplicate both fields.

Given that QUIC doesn't have its own length field, I would imagine it relies on that of UDP, at least in situations where the lower layer does not provide reliable framing?

> Imagine if instead of TCP, SCTP had won the protocol wars in 1980-1990s. [...] We could have had the opportunity to experiment with different types of transports, baked encryption in at a lower level, etc.

Would we? Instead of TCP, SCTP would have become ossified in the stack all other things being equal (in particular the need for something like NAT due to IPv6 exhaustion), no?

[Hydraulix989]

Not sure I follow. QUIC is just UDP with an extra header in the opaque payload. To a firewall, it looks just like UDP.

[jabiko]

That's the point he is making. QUIC has to be based on UDP because the networking stack is ossified enough to not allow the addition of any new Layer 4 protocol. It's not a huge drawback though.

[Hydraulix989]

It's an eight byte penalty for things needed for QUIC anyways.

[sedawkgrep]

I think they’re saying that due to how firewalls are deployed, everything end up either being built on tcp or udp, instead of using existing (or building new) layer four protocols more suited to solving the problem like sctp, et al.

I’m not sure I agree though, because many firewalls already pass other protocols today, like GRE, IPSEC, etc.

[IgorPartola]

That is exactly my point, thank you for clarifying. And yes IPSEC had started forcing people to open up their firewalls. If I had my way though it would be the other way around: all IP protocol numbers except those specifically deemed obsolete or insecure should be allowed, including a range for user defined custom protocols. We really painted ourselves into a corner of 6s and 17s.

[p_l]

It exists partially because of ossification of protocols that killed SCTP.

As in, we wouldn't have to implement it on top of UDP

[jlokier]

Their point is that's a sad design choice, caused by firewalls forcing QUIC to take a first-class TCP-like internet protocol and wrap it with encrypted headers inside UDP for the sole purpose of preventing firewalls from blocking it or making it break in myriad subtle ways. Even QUICs unencrypted header parts are designed to be difficult for intermediate network equipment to track and modify, because of their history of making other protocols over TCP unreliable by doing this.

SCTP is another high-performance protocol that provides many of the same network and performance features as QUIC, but it has been around for ~25 years. It was widely implemented. It was even in Linux 2.4 and available for Windows XP and MacOS. WebRTC in browsers can use SCTP, which makes sense (it predates QUIC) but is actually a bit of a problem if you want wide interoperability with WebRTC.

But despite SCTP being around and widely implemented, and doing many of the good things QUIC does, SCTP was never widely adopted, nor seriously considered for HTTP, mainly because of the problem of firewalls. (And to be fair, some operating system API limitations).

Basically, if TCP were invented today, it could not be deployed succesfully over much of the internet unless it was wrapped in UDP and encrypted.

UDP-with-encryption is now playing the role that the IP protocol is supposed to play, just because too many firewalls block nearly every IP packet that doesn't have type TCP or UDP. If UDP is used but without encryption for a popular protocol, eventually too many firewalls deep-packet-inspect that UDP protocol too, and do the same kinds of blocking, tampering or making protocols break.

This sad problem is called protocol ossification. As Google's AI puts it: "QUIC is the first IETF transport protocol to deliberately minimise its wire image to avoid ossification.". See also https://en.wikipedia.org/wiki/Protocol_ossification and https://http3-explained.haxx.se/en/why-quic/why-ossification

There are some downsides to QUIC's ossification resistance. I did some work for a mobile equipment manufacter who adaptively manage the queues for multiple TCP streams from different users and different applications from the same user, to help ensure a better, fairer and lower-latency experience. QUICs properties made it difficult to measure what's going on at the protocol level, forcing the queue management to use crude statistical estimators instead, which don't work well on bursty protocols.

[tptacek]

UDP is playing exactly the role it was intended to play. Apart from saving 8 bytes of header and a checksum, what's the advantage to running the protocol directly on top of IP? How is this "ossification"? Again: this was David Reed's original design purpose for UDP! It was exactly for doing shit on top of IP without having to run through TCP's mechanism.

[Hydraulix989]

Finally some sense in this thread.

[IgorPartola]

Couldn’t have said it better myself.