[jamra]

That seems like a job for sharded databases and caches.

[vasilia]

Of course, we have both, but 100k is nothing if it's not a CDN server which stores static file in-memory. Moreover, the main metric is latency, not a number of connections. You can scale a number of connections with an L3/L4 load balancer, but not latency.