Connections must have unique IP:Port pairs between client and server. You're limited to 65K concurrent connections for the same client. In practice, no one is opening that many connections from a single client.
As such, the load balancer itself can probably hold a group of source IPs to use as second-hop solution to this problem as well if we're sincerely talking about load balancers holding a ton of largely idle connections simultaneously.
The more likely load balancer outcome would be DNS split on inbound client IPs, and scaling out until each load balancer handles the appropriate amount of traffic (by some measure and scale out if exceeded).