[lvl155]

My reason for going Epyc was for Pcie lanes and cheaper enterprise SSDs via U.3/2. With AM5, you tap out the lanes with dual GPUs. Threadripper is preferable but Epyc is about 1/2 of the price or even better if you go last gen.

[Eisenstein]

Why do you need such high cross card bandwidth for inference? Are you hosting for a lot of users at once?

[oceanplexian]

The Epyc boards make things way easier (I have 4 epyc boards of various generations) because they have loads of x16 slots and you’re not screwing around with bifurcation and sketchy PCI splitters. Another oft-forgotten item that consumes lanes is 25 or 40Gb NICs which you might fine you want if you’re pushing big model files around to other machines or storage.

[darknoon]

I tried this w/ AM5, but realized that despite there theoretically being enough lanes for dual x16 PCI-e 4.0 GPUs, I couldn't find any motherboards that are actually configured this way, since dual-GPU is dead in consumer for gaming.