[yvdriess]

Communication is a key bottleneck. Modern supercomputers budget roughly half to interconnect. Nvidia invested heavily in developing NVLINK to be able to make GPUs communicate at the bandwidth necessary to make multi-GPU boxes more practical (e.g. DGX-1). To make it scale out across multiple boxes, you need a high bandwidth, low latency interconnect, which is where IB comes in.