[yobanate]

Can confirm. My M3 Max gets about 22t/s, putting the bottleneck BKAC.

[3abiton]

That's 10x speed increase. What's the secret behind apple M3? Faster clocked RAMs? Specific AI hardware?

[bugglebeetle]

Unified memory and optimizations in llama.cpp (which Ollama wraps).

[ithkuil]

Is that using the GPU?

[bugglebeetle]

It can be variably configured. There are details in the repo, but llama.cpp makes use of Metal.