Hacker News new | ask | show | jobs
by ChuckMcM 3821 days ago
When I see things like "data set size 150GB" and "1000 CPUS" I just naturally assume they are all in memory and never come from disk :-)
2 comments

That's one of many data sets on the server, so unfortunately we can't keep them all in memory at once. :(
Lets assume when you're saying "cpu" when you mean "core" and your typical server class machine has 24 of those. A 1000 "cpus" is 41 machines, if they each donate 32GB to the cause[1] that is 1.3TB worth of data which is only a few microseconds away from any core.

I'm not sure why anyone would build a server with less than 96GB on it these days, so its not at all unreasonable. Now your service provider my jerk you around but you can run two racks of machines (48 machines) in a data center with specs like that for about $25K/month (including dual gigabit network pipes to your favorite IP transit provider) So it isn't even all that huge of an investment.

[1] Consider your typical 'memcached' type service where data is named as a function of IP and offset.

I think that data set is too small to constitute a good benchmark for the setup.
You're not wrong, but apparently such a short burst is what they're actually doing in their application.