[hliyan]

That just reminded me of the most mysterious scaling issue I ever faced. We had a message to disseminate market data for multiple markets (e.g. IBM: 100/100.12 @ NYSE, 101/102 @ NASDAQ etc.). The system performed admirably under load testing (think 50,000 messages per second). One day we onboarded a single new regional exchange and the whole market data load test collapsed. We searched high and low for days without success, until someone figured out that the new market addition had caused the market data message to exceed the Ethernet frame size for the first time. Problem was not at the application layer or the transport, it was data link layer fragmentation! Figuring that out felt like solving a murder mystery (I wasn't the one who figured it out though).

[jkhdigital]

Classic example of a leaky abstraction, and the principle that implementation details inevitably become undocumented API behavior.

[kabdib]

A lot of "transparent RPC" systems are like this. "It's just like a normal function call, it's sooo convenient" . . . until it isn't, because it involves the network hardware and configuration, routing environment, firewalls, equipment failure . . .

[andylynch]

I’ve worked on systems like this too - the max packet size is very well documented. Then post trade it all gets turned into FIXML which somehow manages to be both more verbose and less readable.

[angstrom]

Yeah, that's part of the trick for large listing responses to be spread across frames. Usually with some indicator like a "more flag" so the client can say "get me the next sequence by requesting the next index in the listup with the prior btree index. People do this all the time with large databases and it's a very similar use case.

[vendiddy]

This was a fun back and forth to read!

[cutemonster]

How did you solve the problem?

[elcritch]

Ouch that’s rough. One nice bit of IPv6 is that it doesn’t allow fragmentation. It often much nicer to get no message or an error than subtly missing data.

[depereo]

IPv6 does allow fragmentation.

[elcritch]

Ah yah that’s right. I’m just learning more of ipv6 and get it mixed up. It appears what I had in my mind was about intermediate routers: “Unlike in IPv4, IPv6 routers (intermediate nodes) never fragment IPv6 packets.” (Wikipedia). To the previous point, it looks like ipv6 does require networks to send 1280 byte or smaller packets unfragmented.