[girvo]

We've not had cheap GPUs with this much VRAM before, though. Might be an interesting change, though I also doubt it personally.

[nl]

As noted in my other comment you can get obsolete GPUs (P100s, BC250s) with lots of RAM on AliExpress now. It hasn't proven revolutionary.

[swiftcoder]

is 16Gb "lots of RAM" in an LLM world? How many of these would you need to stack on a motherboard to inference a decent size model?

[oblio]

At least 4, probably 6+.

[fc417fc802]

BC250 shares memory with system. 16 GB GPUs have long been available to consumers for a small premium. What's never been readily available before is 40+ GB of HBM.

[girvo]

Those don't have a lot of RAM though. Not like these 40, 80GB ones.