[Flowdalic]

The problem does not seem to be that TCP_NODELAY is on, but that the packets are sent carry only 50 bytes of payload. If you send a large file, then I would expect that you invoke send() with page-sized buffers. This should give the TCP stack enough opportunity to fill the packets with an reasonable amount of payload, even in the absence of Nagel's algorithm. Or am I missing something?

[kevincox]

Even if the application is making 50 byte sends why aren't these getting coalesced once the socket's buffer is full? I understand that Nagle's algorithm will send the first couple packets "eagerly" but I would have expected that onced the transmit window is full they start getting coalesced since they are being buffered anyways.

Disabling Nagle's algorithm should be trading network usage for latency. But it shouldn't reduce throughput.

[Flowdalic]

> Even if the application is making 50 byte sends why aren't these getting coalesced once the socket's buffer is full?

Because maybe the 50 bytes are latency sensitive and need to be at the recipient as soon as possible?

> I understand that Nagle's algorithm will send the first couple packets "eagerly" […] Disabling Nagle's algorithm should be trading network usage for latency

No, Nagle's algorithm will delay outgoing TCP packets in the hope that more data will be provided to the TCP connection, that can be shoved into the delayed packet.

The issue here is not Go's default setting of TCP_NODELAY. There is an use case for TCP_NODELAY. Just like there is a use case for disabling TCP_NODELAY, i.e., Nagle's algorithm (see RFC 869). So any discussion about the default behavior appears to be pointless.

Instead, I believe the application or a underlying library is to blame. Because I don’t see why applications performing a bulk transfer of data by using “small” (a few bytes) write is anything but a bad design. Not writing large (e.g., page-sized) chunks of data into the file descriptor of the socket, especially when you know that there multiple more of this chunks are to come, just kills performance on multiple levels.

If I understand the situation the blog post describes correctly, then git-lfs is sending a large (50 MiB?) file in 50 bytes chunks. I suspect this is because git-lfs (or something between git-lfs and the Linux socket, e.g., a library) issues writes to the socket with 50 bytes of data from the file.

[kevincox]

> Because maybe the 50 bytes are latency sensitive and need to be at the recipient as soon as possible?

The difference in latency between a 50 byte and 1500 byte packet is miniscule. If you have the data available in the socket buffer I don't see why you wouldn't want to send it in a single packet.

The latency benefit of TCP_NODELAY should be that it isn't waiting for user space to write more data, not that it is sending short packets.