Hacker News new | ask | show | jobs
by chubot 4276 days ago
FWIW, in 2011, Google wrote that they achieved a PB sort in 33 minutes on 8000 computers, vs. 234 minutes on 190 computers with 6080 cores reported by Spark here.

http://googleresearch.blogspot.com/2011/09/sorting-petabytes...

1 comments

I'm not sure why you list Google as using "8000 computers" and Spark using "190 computers with 6080 cores".

Using two different metrics for two like things seems like there is some sort of implication there. Were Google's machines single-cored?

I'm just writing down exactly what they reported. They used different metrics.

Certainly it would be interesting to have an apples to apples comparison. But the computers aren't the only thing that is relevant -- we also need to know about the networking hardware.