[jeroenhd]

You don't really need 77 IP addresses (the 64k limit for TCP is per client IP, per source port, per server IP) but even if you did, your average IPv6 server will have a few billion available. Every client can connect to a server IP of their own if you ignore the practical limits of the network acceleration and driver stack. If you're somehow dealing with this scale, I doubt you'll be stuck with pure legacy IP addressing.

The real problem with such a setup is that you're not left with a whole lot of bandwidth per connection, even if you ignore things like packet loss and retransmits mucking up the connections. Most VPS servers have a 1gbps connection, with 5 million clients that leaves 200 bytes per second of concurrent bandwidth for TCP signaling and data to flow through. You'll need a ridiculous network card for a single server to deal with such a load, in the terabits per second range.