[tjgq]

SCTP would be the ideal solution, but it's essentially broken on the wide Internet since virtually no NAT boxes understand it.

Using a large number of TCP connections breaks some of the assumptions behind TCP congestion control; this is one of the reasons why the SPDY/HTTP2 guys decided to layer a multi-stream protocol on top of a single TCP connection - and it still falls prey to head-of-line blocking.

Not sure what you mean by using an internal work queue - short of running a custom TCP/IP stack in kernel space or using some non-standard socket API, there's no way to work around TCP's ordered delivery.

Given these constraints, I can understand why designing a transport protocol on top of UDP might be reasonable. There probably is no need to roll your own, though - just use SCTP over UDP [1]. (Edit: I only noticed you did mention SCTP/UDP after I wrote my reply, sorry!)

[1] https://tools.ietf.org/html/rfc6951

[colanderman]

SCTP would be the ideal solution, but it's essentially broken on the wide Internet since virtually no NAT boxes understand it.

Hence my suggestion of SCTP over UDP.

Using a large number of TCP connections breaks some of the assumptions behind TCP congestion control

Which assumptions? Using multiple connections is pretty standard in the enterprise storage world.

Not sure what you mean by using an internal work queue - short of running a custom TCP/IP stack in kernel space or using some non-standard socket API, there's no way to work around TCP's ordered delivery.

My bad, I missed that you were referring to lost packets blocking future packets (as opposed to slow-to-process work units blocking future work units). You're right that this is very much a problem with a single TCP stream. (Multiple TCP streams help with this.)

[tjgq]

> Which assumptions? Using multiple connections is pretty standard in the enterprise storage world.

TCP congestion control is designed such that, under steady-state conditions, each TCP stream traversing a physical link gets an equal share of the available bandwidth. When an application that uses multiple TCP connections competes with another that uses a single one over the same link, the resulting bandwidth distribution is unfair to the latter. (Edit: there's also another problem, related to bufferbloat, but that one is a lot trickier to explain).

Of course, this may not be a problem when used on an internal network (which I assume is the scenario you are referring to). On the public Internet, however, HTTP is becoming increasingly disruptive to other kinds of traffic due to the increasing number of parallel connections used by web browsers.

[colanderman]

Ah yes. I'd say that's more a problem with TCP in general than in using multiple connections. The assumption that "1 TCP connection == 1 share of bandwidth" is at best a useful first approximation. I don't know that N TCP streams eating N times their "fair share" is really any different a problem than some-important-interactive-application being given equal bandwidth with some-irrelevant-background-download. (Though it might be a worse problem.)

I'd love to live to see the day that something actually better than TCP (that addresses these and other issues) dethrones it, but given how long IPv6 took to gain traction, I wouldn't be surprised if it didn't happen in my lifetime.

[tjgq]

I think it's a wonderful thing that the TCP/IP model (as originally envisioned) does not preclude endpoints from implementing their own transport protocols on top of IP, irrespective of what the rest of the network uses.

Unfortunately, reality has conspired against this possibility in two ways: NAT boxes, and the fact that transport protocols are implemented in the kernel on most operating systems (hence, an application cannot use its own transport protocol easily).

The deployment of IPv6 is a real chance to, at least, fix the first of these problems. Let's hope it does.