[lovesdogsnsnow]

Aren't interconnect bandwidth and latency important for training workloads?

[cavisne]

Yes, but to move the needle you need to use another single vendor Mellanox (which nvidia owns).

So the (slight) differentiations are.

AWS - gpudirect and high bandwidth (but also high latency)

GCP - 16 gpus on one machine

Azure/OCI - mellanox