[Veserv]

They are proposing custom hardware on both ends talking a custom hardware network protocol. That is a enormous amount of complexity in comparison to a custom software stack on bog-standard hardware. I would expect advantages to justify that level of complexity.

However, people like yourself talk about these hardware stacks as if they have clear advantages in performance and latency and isolation. They make uncurious and dismissive comments without evidence that this level of results is only achievable with dedicated hardware.

The only consistent conclusion I can come up with is that everybody just uses really bad software stacks which makes these dedicated hardware solutions seem like major improvements when they are just demonstrating performance you should expect out of your software stack. The fact that this is considered a serious improvement over RoCE which is itself viewed as a serious improvement over things like the Linux kernel TCP software stack lends support for my conclusion.

I make comments on various posts about network protocols to see if I am missing something about the problem space that actually makes it hard to do efficiently in a software protocol. Mostly I just get people parroting the claim that a performant software solution is impossible due to easily solved problems like loss/recovery/retransmission instead of actually indicating hard parts of the problem.

And as for what would be useful hardware I would go with a network with full 64K MTU and hardware copy offload with HBM or other fast bus. Then you could pretty comfortably drive ~10 Tb/s per core subject to enough memory/bus bandwidth.

[jauntywundrkind]

It feels like you're not really trying, if you can't cite anything in genre with what you are talking about?

I'd love to run off Google Falcon versus say Microsoft's Machnet. There's a ton of system resources dedicated to making machnet and dpdk fast, it comes with huge design tradeoffs & specially carefully crafted software architectures. Extreme software engineering effort. The glory of Falcon is that you don't need to rebuild your app stack from the bottom up like this: you're just going to get blitz fast insane utilization with incredibly low p99 for free from the system you have. https://github.com/microsoft/machnet

I think you massively over-glorify the software side of things. There is a ton of potential absolutely to very carefully setup systems to get great goodput, no degredation. But it usually requires carefully built systems & eternal vigilance in how you scale your software across cores. Or you could just have some hardware that routes replies directly to the place it needs to go. I don't get what feels like a weird fetishization you have with doing it in software, and it feels like you don't have any rigor about your criticism/can't cite any backing research/want to accuse everyone else of being super lazy.