[PlatoIsADisease]

You might want to clarify that this is more of a "Look it technically works"

Not a "I actually use this"

The difference between waiting 20 minutes to answer the prompt '1+1='

and actually using it for something useful is massive here. I wonder where this idea of running AI on CPU comes from. Was it Apple astroturfing? Was it Apple fanboys? I don't see people wasting time on non-Apple CPUs. (Although, I did do this for a 7B model)

[mholm]

The reason Macs get recommended is the unified memory, which is usable as VRAM for the GPU. People are similarly using the AMD Strix Halo for AI which also has a similar memory architecture. Time to first token for something like '1+1=' would be seconds, and then you'd be getting ~20 tokens per second, which is absolutely plenty fast for regular use. Token/s slows down at the higher end of context, but it's absolutely still practical for a lot of usecases. Though I agree that agentic coding, especially over large projects, would likely get too slow to be practical.

[PlatoIsADisease]

We are getting into a debate between particulars and universals. To call the 'unified memory' VRAM is quite a generalization. Whatever the case, we can tell from stock prices that whatever this VRAM is, its nothing compared to NVIDIA.

Anyway, we were trying to run a 70B model on a macbook(can't remember which M model) at a fortune 20 company, it never became practical. We were trying to compare strings of character length ~200. It was like 400-ish characters plus a pre-prompt.

I can't imagine this being reasonable on a 1T model, let alone the 400B models of deepseek and LLAMA.

[Gracana]

With 32B active parameters, Kimi K2.5 will run faster than your 70B model.

[simonw]

Here's a video of a previous 1T K2 model running using MLX on a a pair of Mac Studios: https://twitter.com/awnihannun/status/1943723599971443134 - performance isn't terrible.

[PlatoIsADisease]

Is there a catch? I was not getting anything like this on a 70B model.

EDIT: oh its a marketing account and the program never finished... who knows the validity.

[simonw]

I don't think Awni should be dismissed as a "marketing account" - they're an engineer at Apple who's been driving the MLX project for a couple of years now, they've earned a lot of respect from me.

[PlatoIsADisease]

Given how secretive Apple is, oh my, its super duper marketing account.

[zozbot234]

Not too slow if you just let it run overnight/in the background. But the biggest draw would be no rate limits whatsoever compared to the big proprietary APIs, especially Claude's. No risk of sudden rugpulls either, and the model will have very consistent performance.

[simonw]

MLX uses the GPU.

That said, I wouldn't necessarily recommend spending $20,000 on a pair of Mac Studios to run models like this. The performance won't be nearly as good as the server-class GPU hardware that hosted models run on.

[tucnak]

Mac studio way is not "AI on CPU," as M2/M4 are complex SoC, that includes a GPU with unified memory access.

[PlatoIsADisease]

If it worked IRL for anything useful, I'd be more interested in the technical differences. But it was a mere toy for a few tests at my fortune 20 company.

Language is full of issues of particulars vs universals, and you could debate if its just an integrated GPU with different marketing.

Whatever the case, we couldn't use it in production, and NVIDIAs stock price reflects the reality on the ground.

[tucnak]

Well, I've been using a fine-tuned variant of Gemma 3 model since it came out, and some embedding models, on a laptop. It's not "useless" by any means, in fact it still beats the latest Claude for my use-case in Ukrainian. Not to mention that if you travel by train a lot, you will find it quite useful. I own a Mac studio M2 Max (96 GB) variant at home, and I'm routinely using the larger models for the kind of stuff I don't wish to share with model providers.

My 2 cents