[rxin]

Author of the blog post here. Kay already pointed out that increasing cluster size usually reduces network utilization. In addition to that, there a few challenges with just getting bigger cluster size with "on-demand" resources:

1. Obviously, it costs more to get a bigger cluster.

2. You might not be able to get as big of a cluster as you wanted, since "cloud" is not an infinite resource.

3. Some workloads are not embarrassing parallel. For some (graph, matrix, join), more parallelism -> more communication.