[davidzweig]

Is it possible a similar patch would work for P2P on 3090s?

btw, I found a Gigabyte board on Taobao that is unlisted on their site: MZF2-AC0, costs $900. 2 socket Epyc and 10 PCIE slots, may be of interest. A case that should fit, with 2x 2000W Great Wall PSUs and PDU is 4050 RMB (https://www.toploong.com/en/4GPU-server-case/644.html). You still need blower GPUs.

[georgehotz]

It should if your 3090s have Resizable BAR support in the VBIOS. AFAIK most card manufacturers released BIOS updates enabling this.

Re: 3090 NVLink, that only allows pairs of cards to be connected. PCIe allows full fabric switch of many cards.

[Ratiofarmings]

In cases where they didn't, the techpowerup vBIOS collection solves the problem.

[learnedmachine]

Thanks for the amazing work! I tried the driver with some 3090s (all of which show the 32G line with lspci -s 01:00.0 -v) and while torch says I have p2p access, I can't get it to work with anything as I get illegal memory access errors.

[davidzweig]

Update, I checked with the case company, toploong, they say that board is a 5mm too big or so for the case.

[cjbprime]

Doesn't nvlink work natively on 3090s? I thought it was only removed (and here re-enabled) in 4090.

[qeternity]

This not not nvlink.