[verditelabs]

I think I recognize the author of this from /r/localllama, where plenty of other people are building similar frankenstein rigs. This post only mentions Intel setups, but AMD Epyc Milan and Rome based rigs are also very viable alternatives. They're a bit more expensive, but much better perf/watt, and the incremental price increase after factoring in a lot of GPUs is fairly slim. With 7 PCIE bifurcators on a motherboard such as the AsRock ROMED8-2T and a 14 risers, you can get up to 14 GPUs at PCIE 4.0x8.

[3abiton]

A bit sad hobbyist have to resort to such measures to get tinkering, not to mention the initial capital needed. We're all slaves to Nvidia's VRAM monopoly until AMD or Intel steps in and release a competitive alternative with beefy vrams.

[loudmax]

Beefy VRAM is a start, but Nvidia's real moat is CUDA. If PyTorch runs on AMD's ROCm, or the Intel equivalent, as well as it runs on CUDA, then we'll see some real competition here.

Chris Lattner's Mojo programming language may present an alternative here, but it's still closed source.

[3abiton]

How will Mojo solve what is essentially a GPU driver issue (ROCm vs cuda)? AMD's track at supporting GPU computing is very poor to be trusted so far.