[Lanzaa]

I believe the network had become a bottleneck. As per the article:

> [O]ur Spark cluster was able to sustain ... 1.1 GB/s/node network activity during the reduce phase, saturating the 10Gbps link available on these machines.

If the network is the bottleneck it makes sense to reduce the number of nodes to reduce the network communications.

[rxin]

The job is actually very linearly scalable. i.e. running it on 200 nodes roughly doubles the throughput of 100 nodes.