[dave168]

CNTK is great at scaling out beyond a simple machine. The paper didn't benchmark that but only tested one single box performance.

[Eridrus]

Realistically, most people barely get to multiple GPUs, let alone multiple machines. You're more likely to do hyperparameter tuning across machines before you do distributed training.