[Rinzler89]

>even with an old 8 GB AMD card

That's the main bummer here, not the age of the GPU, but the need for a dedicated one.

When most people nowadays have laptops, and only with integrated GPUs to boot, that's a large demographic being excluded, even with new machines.

Maybe with the new gen of chips with NPUs from Qualcomm, Intel and AMD this will slowly change but they just dropped and I personally won't exchange my 9 month old laptop for a new one just for that feature alone, and I suspect most PC users will behave the same considering the multi year long upgrade cycles in this market.

So it will probably take a long time till on-device generative AI becomes mainstream on the PC.

[numpad0]

That's where Qualcomm/Intel/Microsoft "AI PC" branding with 40 TOPS bar comes in. Besides, GPU had been a requirement for AI for years by now.

[mft_]

For decent on-device inference, you need enough memory of high-enough bandwidth and enough processing power connected to that memory.

The traditional PC architecture is ill-suited to this, which is why for most computers, a GPU (which offers all three in a single package) is currently the best approach... only at the moment, even a 4090 only offers enough memory for moderately-sized models to load and run.

The architecture that supports Apple's ARM computers is (by design or happenstance) far better suited: unified memory maximises the model that can be loaded, some higher-end options have decently-high memory bandwidth, and the architecture lets any of the different processing units access the unified memory. Their weakness is cost, and that the processors aren't powerful enough yet to compete at the top end.

So there's currently an empty middle-ground to be won, and it's interesting to watch and wait to see how it will be won. e.g.

- affordable GPUs with much larger memory for the big models? (i.e. GPUs, but optimised)

- affordable unified memory computers with more processing power (i.e. Apple's approach, but optimised)

- something else (probably on the software side) making larger model inference more efficient, either from a processing side (i.e. faster for the same output) or from a memory utilisation side (e.g. loading or streaming parts of large models to cope with smaller device memory, or smaller models for the same output)