[mathisfun123]

> I'm curious and not an expert here, do you know why the TTFT is so much worse on Mac?

because the GPUs aren't as fantastic as everyone assumes?

> might also be less optimised in MLX?

prefill has gotta be one of the most optimized paths in MLX...

[bigyabai]

No you don't understand, on Apple Silicon my CPU has comparable memory bandwidth to a $400 Pascal-era GPU. With the unified memory architecture, that means my iGPU gets 2016-levels of DDR transfer speed with none of the upsides of CUDA. It's the most cutting-edge hardware ever put in a personal computer, without a doubt.

[fgfarben]

Please show me on the 2016-era $400 Pascal GPU where you can install the 256 GB of VRAM.

[bigyabai]

We're quite lucky that Nvidia didn't ship a 256gb system at sub-500gb/s transfer rate, is my point.

[fgfarben]

> Nvidia didn't ship a 256gb system at sub-500gb/s transfer rate

DGX Spark has 128 GB and only 273 GB/s BW. Are we lucky that NVIDIA did ship something even worse than what you specified? I'm confused.

People have been complaining [1] about how little VRAM NVIDIA ships with their GPUs for decades. Their whole game has been "oh, you want more VRAM? Buy more or pay us 50x for server grade with 10x as much VRAM. The more you buy, the more you save."

Apple did everyone a solid by shipping something way out of that distribution. We now know more than we did before! We know that a 284B parameter model with 13B active params (or 35B with 3B active, or 671B with 37B active) can outperform a 2T model and draw a fraction as much power. How can you think that's a bad thing?

You could point out that Apple didn't invent the idea of MoE. Everyone knows that. But other than Macs, there simply were no machines with >100GB VRAM directly coupled to ~50 TFLOP/s of compute until the DGX Spark last Dec. If you wanted to run a model with more than 32 GB of weights, you had to either pay up for dozens of GPUs idling at hundreds of watts or really pay up for some $50,000 server GPUs idling at... also 100-200W each.

I feel lucky to have a $3k machine on my shelf that can run DS4-Flash with 1M context at 20t/s while drawing ~150W and making very little noise. The best part? It idles at 30W with DS4 loaded, dropping to 6W after a reboot. There isn't a single GPU on the market that can match that in the same shoebox volume.

[1] https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRlOW0N...

[bigyabai]

The DGX Spark is also a niche, arbitrarily limited machine that will not displace serious datacenter workloads. It's targeted directly at the homelab LARPers and arguably a waste of money versus similarly priced GPU clusters. A 256gb Spark at LPDDRX5 transfer rates would be a genuine travesty.

You can try to weasel out any sort of edge justification you want - these are not industry-grade machines. They are slow, expensive, bandwidth-constrained SOCs that don't hold a candle to either datacenter GPUs or even decade-old gaming GPUs. It's worth criticizing when Apple does it, and also worth criticism when Nvidia does it. The only difference being that Nvidia has natural datacenter buy-in, while Apple can't even justify their own hardware in the face of TPU inference costs: https://9to5mac.com/2026/03/02/some-apple-ai-servers-are-rep...

[fgfarben]

What even is an industry grade machine?

Would you own a computer if the smallest computer you were allowed to buy was a $27,000 Supermicro rack that draws 900 watts all the time?