[make3]

how does the problem fair on gpus

[hyperbovine]

If it requires synchronizing across workers, poorly.

[make3]

depends how often you need to synchronize. look at deep learning, you need to sync the weights as often as possible

[malux85]

This is correct