[ftufek]

Yeah, it's really just trying to get as much compute as possible as cheaply as possible interconnected in a reasonably fast way with low latency. Slow networking would be a bottleneck and expensive high end networking would defeat the purpose of staying cheap.

[buildbot]

You’d be surprised at how cheap high end networking that outperforms PCIE4 x4 is - 100Gb omni-path nics are running for 20$ on ebay! And those will saturate PCIE3 x16.

Though of course with multiple boards/ram/cpu it gets complicated again.

[magicalhippo]

Which cards? Been looking at nics but couldn't find cheap ones past 25-40Gb

[justinclift]

Omni-path is/was the Intel fork of Infiniband, which from rough memory they bought from QLogic some years ago.

* Switch: https://www.ebay.com/itm/273064154224

* Adapters: https://www.ebay.com/itm/296188773061 / https://www.ebay.com/itm/166199451199

* Cables: No idea, sorry. ;)

* Description: https://www.youtube.com/watch?v=dOIXtsjJMYE

Note that I don't know those Ebay sellers at all, they're just some of the cheaper results showing up from searching. There seem to be plenty of other results too. :)

---

Possibly useful, though it's Proxmox focused:

https://forum.level1techs.com/t/proxmox-with-intel-omni-path...

[jacquesm]

Very smart approach. I may copy your setup for some project that I've been working on for years but that stalled waiting for more memory in GPUs.