[literalAardvark]

The point of distributed computing is to do computing that you can't do on a vertically scaled system or to increase availability.

If you're doing it for other reasons it's usually a mistake.

[raddan]

The advice I’ve gotten is that you want to move computation to data that is already distributed. The cost of moving large amounts of data usually dwarfs compute costs (usually, not always), and so the performance win comes from distributing the computation and then (depending on the problem) centralizing aggregate results.

[subhobroto]

Another pretty good reason to do distributed computing is to move the computation closer to where the data is or where the data will be consumed.