[DanielHB]

I heard nvidia is gimping consumer-grade cards to not be good at LLM training, is this true? If so are they gimped only for training or also for running LLMs?

I guess the limited amount of RAM is also a way to limit the cards.

[kimixa]

Many Nvidia "gaming" SKUs are already at the point where memory is often the biggest likely limitation on their gaming use case, and they'll be noticeably better products for the consumer with a small cost increase by adding more memory.

So I'd say there's good evidence that something outside cost and value to the gaming use case is why they don't have higher memory SKUs, and eating into "professional" priced AI SKUs is an obvious possibility.

I doubt anyone outside Nvidia itself knows "for sure", but it's a pretty big indication.

[mcraiha]

At least Mistral 7B for its 128 token text generation is 58% faster with 5090 compared to 4090. https://www.phoronix.com/review/nvidia-rtx5090-llama-cpp/3