[mmt]

> We run 30 node, autoscaling clusters which stay close to 100% for most of the time.

I have exactly zero knowledge on Spark's efficiency as well as zero on how representative graph algorithms are, but I can confidently say that the above statement fails to refute the referenced article's thesis (which, arguably, criticizes assertions just like that).

Just because your implementation scales (even autoscales) to use more compute resources says nothing about its efficiency (overall or even marginal when adding more nodes, i.e. the shape of the curve).

Computer science has struggled with achieving even near linear-scalability ever since the advent of SMP.