Hacker News new | ask | show | jobs
by barbariangrunge 1138 days ago
Yeah. It seems to me that it's really hard to get more than 10-14 GB of VRAM without using some sort of hyper expensive cluster. What would it cost if you wanted to do it with Nvidia? Being able to share ordinary ram with the GPU in a Mac could maybe be a unique value proposition
2 comments
int_19h 1137 days ago
RTX 3090 or 4090 gets you 24Gb of VRAM, which is enough to run llama-30b (quantized to 4-bit with groupsize of 1024 or higher) at speeds comparable to ChatGPT. You can also get two and run the model split across them, although pumping data back and forth slows things down.

A brand new RTX A6000 (48Gb VRAM) is probably the largest you can get in a single card that can run in a regular PC. It can be had for $4-5k and is sufficient for llama-65b.

Beyond that, yeah, you're looking at dedicated multi-GPU server hardware.

link
dragonwriter 1137 days ago
> It seems to me that it’s really hard to get more than 10-14 GB of VRAM without using some sort of hyper expensive cluster.

Both consumer and workstation (the latter may be cheaper per RAM, but with fewer shaders) 16-24 GB GPUs (RTX 3080Ti/3090/4090/A4000/A4500/A5000), including in laptops, are not hard to find (pricey, but not “hyperexpensive clusters”), and its not until you jump above a single 48 GB RTX A6000 that you need a “cluster”.

link