[wtallis]

A combination of high memory bandwidth and large memory capacity is necessary for good performance on LLMs. Plenty of consumer GPUs have great memory bandwidth but not enough capacity for the good LLMs. AMD's Phoenix has a memory bus too narrow to enable GPU-like bandwidth, and when paired with the faster memory it supports (LPDDR5 rather than DDR5) it won't offer much more memory capacity than consumer GPUs.

[diffeomorphism]

> won't offer much more memory

A mini PC with that chip, 1 TB of storage and 64GB of ram (both replaceable) costs like 800€ and fits behind your monitor. Getting that much memory in a consumer GPU is definitely quite a bit more expensive. Also, for comparison an M2 Ultra with that amount of storage and ram is 4800€.

So I am not doubting that a 6 times as expensive computer is probably "better" by some metric, but for that drastic difference I am not sure that is enough.

[moooo99]

While I 100% agree on the price comparison, you’ll need to reach some threshold for LLM performance to consider them as usable. As Someone not very knowledgeable at the topic, the pure difference in the numbers lead me to question if you could even reach that usable performance threshold with the 800€ mini PC

[wtallis]

Note that when referring to memory capacity, I specified LPDDR5, because that's the fastest memory option. If you want to go with 64GB of replaceable DDR5, you'll sacrifice at least 18% of the memory bandwidth. (And in theory the SoC supports LPDDR5-7500, but I'm not aware of anyone shipping it with faster than LPDDR5-6400 yet.) So you could get to 64GB on the memory capacity with a Phoenix SoC, but only by being at a 10x disadvantage on bandwidth relative to an M2 Ultra—which doesn't make a 6x price difference sound outrageous, given that we're discussing workloads that actually benefit from ample memory bandwidth.