[rdos]

There is no point in using a low-bandwidth card like the B50 for AI. Attempting to use 2x or 4x cards to load a real model will result in poor performance and low generation speed. If you don’t need a larger model, use a 3060 or 2x 3060, and you’ll get significantly better performance than the B50—so much better that the higher power consumption won’t matter (70W vs. 170W for a single card). Higher VRAM wont make the card 'better for AI'.

[bsder]

> There is no point in using a low-bandwidth card like the B50 for AI.

People actually use loaded out M-series macs for some forms of AI training. So, total memory does seem to matter in certain cases.

[robotnikman]

>2x 3060

Are there any performance bottlenecks with using 2 cards instead of a single card? I don't think any one the consumer Nvidia cards use NVlink anymore, or at least they haven't for a while now.

[vid]

Who said anything about the B50?

Plenty of people use eg 2, 4 or 6 3090s to run large models at acceptable speeds.

Higher VRAM at decent (much faster than DDR5) speeds will make cards better for AI.

[hadlock]

If VRAM is ~$10/gb I suspect people paying $450 for a 12GB card would be happy to pay $1200 for a 64gb card. Running local LLM only uses about 3-6% of my GPU's capability, but all of it's VRAM. Local LLM has no need for 6 3090s to serve a single or handful of users; they just need the VRAM to run the model locally.

[vid]

Exactly. People would be thrilled with a $1200 64GB card with ok processing power and transfer speed. It's a bit of a mystery why it doesn't exist. Intel is enabling vendors to 'glue' two 24GB cards together for a $1200 list price 48GB card, but it's a frankenstein monster and will probably not be available for that price.

[wqaatwt]

Nvidia has zero incentives to undercut their enterprise GPUs by adding more RAM to “cheap” consumer cards like the 5090.

Intel and even AMD can’t compete or aren’t bothering. I guess we’ll see how the glued 48GB B60 will do, but that’s a still relatively slow GPU regardless of memory. Might be quite competitive with Macs, though.