[zdw]

Compared to 2-4TB max in a 2 socket server system, the 16 or 32GB/card you get on a GPU is pretty small.

The other issue is loading that memory - PCIe 4 is still the transfer time bottleneck between GPU and main memory.

[atq2119]

The part about no virtual memory is kind of incorrect though. It doesn't work exactly like on the CPU, but still.

[zdw]

I'm not totally up to speed on modern GPU's - do they have a MMU that handles page faults on non-resident pages like a CPU would?

My understanding is that the GPU memory models are different enough that what an OS traditionally calls "virtual memory" couldn't be implemented in the same way.