[gorkish]

If your storage devices are native PCIe and your network is actually fast enough and your nics and switches have the proper hardware and driver support and your switches are properly configured and your nfs client and server configurations are tweaked just so, and you hold your face in just the right position

The perf is amazing

[dlcarrier]

I do have stable diffusion running on a non-Nvidia GPU, so I'm familiar with the manner of rituals required before contemplating installing system updates or even suspending the system.

Now, I'm off to determine if an eMMC M.2 module is a native PCIe device…

[olavgg]

ROCE do require proper nic/switch tuning. But if you go all in on Mellanox NIC's and switches, its basically on knob on the switch: roce

If you buy CX3 and Mellanox SX6036(easy fan adjustment and low power usage) switches on Ebay for cheap. Which are great up to 56Gbps. You can run infiniband which is way easier to setup, just remember you still need to enable IP over IB to start the handshake for the RDMA connection.