[nunez]

They've been investing in NPUs for a long long time. Since the first iPad Pro at least (2018). Apple Silicon is a natural fit for tensor/matrix operations due to unified memory, so if anything them going "all in" on that has been a nice side effect.

[_davide_]

> Apple Silicon is a natural fit for tensor/matrix operations due to unified memory,

This is a common misconception. This is NOT true: memory designed for CPU performs horribly for GPU tasks. Mac/Strix Halo/NVIDIA Spark are performing horribly compared to a desktop video card with GDDR.

I spent quite a few weekends optimizing ROCM kernels for strix halo, i WISH I had GDDR instead of high-frequency CPU RAM; the bandwidth would be SO MUCH better.

I'm quantizing weights not to make computation faster, not because I'm out of memory, but because memory cannot move fast enough to be processed and it's cheaper to load quantized version, convert into bf16 compute, and discard; This happens every single time a token is generated for the whole model over and over.

[TimByte]

The whole point of Apple silicon isnt speed, it is capacity. You can grab a Mac Studio with 192 gb of unified memory and shove a model in there that would otherwise require building a rig with multiple rtx 4090 s. For local dev just being able to fit the weights in memory is often way more important than token generation speed