[htrp]

or if the internet became significantly faster fiber connections

[HPsquared]

A single GPU has memory bandwidth around 1000 GB/s ... that's a lot of fiber! (EDIT: although the PCIE interconnect isn't as fast, of course. NVLink is pretty fast though which is the sort of thing you'd be using in a large system)

[brookst]

Latency still matters a lot…