Local models running on Apple hardware are definitely a thing, a low-hanging fruit they are harvesting right now while every pure-play AI company is losing money.
Ironically for home-lab users (ie those who primarily have access to consumer hardware), apple silicon is probably the best value (and also the most obtainable for running some of the smaller sized models. You can get unified memory of up to 192 GB in the mac studios. The equivalent for the "open market" is probably AMD's halo strix which offers up to 128GB unified memory, and the speed of the memory is much slower.
I guess the Nvidia Jetson AGX Thor also exists, but good luck getting your hands on that... and regardless that's also 128GB.
Put it this way, with 192 GB VRAM you can run deepseekv4... with 128gb only with lots of quantization
Edit: WTH! I was curious and i went to apple's website to price out a high end unit just to price comparison, and it looks like the high ram units are gone now, and they max out at 96GB?? Guess i missed that...
It's not a huge thing outside the consumer/novelty sphere. Apple missed the train on the trillion-dollar ARM datacenter rollout, and will suck those lemons for the next decade.
I guess the Nvidia Jetson AGX Thor also exists, but good luck getting your hands on that... and regardless that's also 128GB.
Put it this way, with 192 GB VRAM you can run deepseekv4... with 128gb only with lots of quantization
Edit: WTH! I was curious and i went to apple's website to price out a high end unit just to price comparison, and it looks like the high ram units are gone now, and they max out at 96GB?? Guess i missed that...