[throwaway_ab]

How would a server/workstation like this be setup?

I thought you could only use the vram on the GPU, so for 700GB you would need 8-9 A100 nodes as 2 only gives 160GB.

I've been trying to figure out how to build a local system to run inference and train on top of LLM models, I thought there was no way to add vram to a system outside of adding more and more GPU's or use system ram (DDR5) even though that would be considerably slower.

[toshinoriyagi]

An A100 node has 8 A100s in it, each with 80GB, which is how they got the 1.28TB number 2 * (80 * 8).