Hacker News new | ask | show | jobs
by Eisenstein 643 days ago
Why do you need such high cross card bandwidth for inference? Are you hosting for a lot of users at once?
1 comments

The Epyc boards make things way easier (I have 4 epyc boards of various generations) because they have loads of x16 slots and you’re not screwing around with bifurcation and sketchy PCI splitters. Another oft-forgotten item that consumes lanes is 25 or 40Gb NICs which you might fine you want if you’re pushing big model files around to other machines or storage.