Hacker News new | ask | show | jobs
by notorandit 1513 days ago
With a maximum of 64k TCP connections per single server IP, you need 77 different IP on the server side. This is a fact.
7 comments

What?

Having run production services that had over 250,000 sockets connecting to a single server port, I'm calling "nope" on that.

Are you thinking of the ephemeral port limit? That's on the client side; not the server side. Each TCP socket pair is a four-tuple of [server IP, server port, client IP, client port]; the uniqueness comes from the client IP/port part in the server case.

You don't really need 77 IP addresses (the 64k limit for TCP is per client IP, per source port, per server IP) but even if you did, your average IPv6 server will have a few billion available. Every client can connect to a server IP of their own if you ignore the practical limits of the network acceleration and driver stack. If you're somehow dealing with this scale, I doubt you'll be stuck with pure legacy IP addressing.

The real problem with such a setup is that you're not left with a whole lot of bandwidth per connection, even if you ignore things like packet loss and retransmits mucking up the connections. Most VPS servers have a 1gbps connection, with 5 million clients that leaves 200 bytes per second of concurrent bandwidth for TCP signaling and data to flow through. You'll need a ridiculous network card for a single server to deal with such a load, in the terabits per second range.

Isn't this limit per client ip, server ip, and server port? (https://stackoverflow.com/a/2332756/303637)
I imagine that's the limit per client IP address [for a single server port], no? The Linux kernel can use multiple pieces of information to track connections: client IP address, client port, server IP address, server port.

Cloudflare has some interesting blog posts on this topic:

- https://blog.cloudflare.com/how-we-built-spectrum/

- https://blog.cloudflare.com/how-to-stop-running-out-of-ephem...

“You need 77 ips” to do what? May be a fact or not, depending on what you’re doing.

If you suppose just one open server port, you’ll probably need 77 client ips to do this test to get unique socket pairs.

But it’s a client problem, not a server one.

How do you figure?

Clients can connect to the server on the same server port, so connection limit is more like 64k*2 for every Client IP-Server IP pair.

Actually every client IP+port / server IP+port pair. Linux uses 60999 − 32768 for ephemeral ports so can support 28e3^2 = 784 million connections per IP pair.
Except your service is almost certainly listening on one non-ephemeral port.

But having "only" tens of thousands of connections per client is rarely a problem in practice, apart from some load testing scenarios (such as the experiment here, where they opened a number of ports so they could test a large number of connections with a single client machine).

1 IP can correspond to multiple different clients.
With NAT, right. I guess there may be situations where client ports for a single server IP dry up due to NAT, but I've not encountered that issue.
Pretty sure you can bump that up in the kernel to hold more active connections per server that 64k...