[davedx]

I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neural networks.

I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a better choice?

[kmacdough]

Its not about geometry, it's a parallel compute thing thing. CPUs typically don't have more than 10 or 20 cores. GPU have 100s to 1000s.

Matmul is very well parallelized. More, lower power cores will always pay off handsomely.

[jboss10]

Most of the time, the speed of these models are constrained by memory bandwidth. GPUs normally have much more memory bandwidth.

[davedx]

I'd expect the memory bandwidth to be the same for the CPU and GPU under a unified memory architecture like Apple silicon uses?