[mpfundstein]

and you didn't even talk about data and model parallelism. which often just works out of the box