Understand, manage, and buy networking equipment for. Infiniband is a thing of the past, especially with the Mellanox VPI adapters that support both Ethernet and Infiniband with a single bit flipped on the adapter.
IDK, IME IB is pretty much plug and play in an HPC setting. Plain ethernet is too, sure, but if you want to do HPC type workloads you'll have to do a lot of configuration and testing to setup DCB, RoCEv2, EVPN+VXLAN+BGP or such.
But I think this is the way the market is going in the longer term.
It is, but if you're a large enough company to be buying millions of dollars in adapters and switches, reading a guide from Mellanox to turn on DCB should be fairly seamless. RoCEv2 is API-compatible with IB for the most part, so there is really no configuration on that layer. The other pieces -- not really sure what you're getting at. Most of those are for going across data centers, which IB won't do anyways. At least Ethernet would give you the option to run RDMA from the east coast to the west coast.
> The other pieces -- not really sure what you're getting at.
What I'm getting at is setting up clusters larger than what you can fit behind a single switch. So you'll want e.g. a CLOS fabric with multipathing (the typical IB setup, FWIW). As Trill and SPB seem pretty dead, it seems the momentum is to do the multipathing at the L3 level, using the aforementioned EPVN+VXLAN+BGP, or something similar.
You mean you have separate subnets for each leaf switch, and then BGP or such for multipath routing between the leaf and spines? Sure, but what about subnet-level services like DHCP & PXE? Sounds cumbersome if you have to replicate that across all your leaf switches?
Or maybe you could do one "provisioning and admin" VLAN that spans the entire cluster and which uses spanning tree, and then the high-performance RDMA stuff uses the per-leaf VLAN's and L3 multipath routing? Is that simpler and better performing that EVPN + VXLAN?
What is the routing latency on such BGP setups BTW? I find it hard to image you can get even close to eth (not to mention IB!) L2 latencies? Or can the fast paths be done in hw (or FPGA's)?
Yes, a subnet per rack is a best practice. Often people DHCP & PXE over the 1G out-of-band network which is dumb L2.
In most ASICs everything is the same latency since packets go through the whole pipeline whether they use all the functionality or not. Anyway, the latency of plain routing would have to be equal or faster than VXLAN encap + routing.
But I think this is the way the market is going in the longer term.