[Jedd]

Related - has anyone done much with UDP load balancing on prem?

We're starting to hit performance and HA walls with ingesting Netflows from edge routers - you can only nominate one target, and using Elasticsearch / Logstash there are some hard limits.

Would AWS be appropriating nginx under the hood here?

[nullwasamistake]

Lots of people use IPVS but the more efficient modes don't work on AWS. Generally why most that need a LOT of traffic use a cloud provider for regular servers and their own servers in CoLo for heavy stuff.

With how Amazon likes to use OSS in their services I'm pretty sure their UDP load balancer are in fact just using IPVS

[colmmacc]

NLB is built on top of AWS HyperPlane, a hybrid system that has components distributed in our Nitro security system, and pseudo-central components that keep flow-tracking state. It's different from IPVS.

[Jedd]

Interesting, thanks. Hadn't considered this option before, and will do some more exploring, though I note on the IPVS page they say:

"For scheduling UDP datagrams, IPVS load balancer records UDP datagram scheduling with configurable timeout, and the default UDP timeout is 300 seconds. Before UDP connection timeouts, all UDP datagrams from the same socket (protocol, ip address and port) will be directed to the same server."

I'm hopeful / confident that affinity can be fully de-tuned here, as we're looking at around 5-10k UDP Netflows per second from a given router that need to be distributed to a set of receivers.

[nullwasamistake]

I may be wrong, but I think you can tell IPVS to schedule using tuple hash only using Direct Return mode, which means no stored state for connection tracking.

Edit: doesn't appear to be true, but it uses it's own "lightweight" connection tracking table so you can unload conntrack modules from kernel.

Realistically IPVS can probably route 40 gigabit of traffic per instance. Combine that with DNS round robin and maybe even multi-homing at the front and you could handle basically anything