[omneity]

I agree with you, that’s how it should be, but that’s not how it currently is.

Looking at what’s available right now. You need 3 A100 40GB to get this amount of VRAM which will cost you way north of 20000$.

Doing it with A6000s is still about 15k$.

There’s not that many high VRAM options out there you know..

[AnthonyMouse]

If all you want is VRAM you can get old P40s with 24GB for $175 and it's 144GB for $1050. Then you need a big machine to put six of them in but that doesn't cost $4000.

But all of this is kludges. The Radeon RX 7900 XTX has more than twice the memory bandwidth of the M3 Max with much better performance per watt than an array of P40s. What you want is that with more VRAM, not any of this misery.

[omneity]

That checks out in principle, but given that P40 doesn't support NVLink, I wouldn't count too much on using six of them together in a performant manner.

But yeah the best option remains an MI300 if you can afford that.

[a_wild_dandan]

Yeah, my M2 MacBook has 96GB @400GB/s. For $4k or so, it feels like cheating. Does it beat 4x24GB NVIDIA cards? Absolutely not! It's slower and occasionally runs into CUDA-moat software issues. But the capability to daily drive Mixtral 8x7 locally, with great token speeds, is phenomenal.

[AnthonyMouse]

NVLink is most needed for training. For inference a lot of the popular models can usefully be run on multiple GPUs without it:

https://www.reddit.com/r/LocalLLaMA/comments/142rm0m/llamacp...

[cyanydeez]

lastgen A6000 is about 5k