[ttul]

Here’s the two main reasons why local inference won’t compete any time soon with the cloud:

1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware.

2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advantages of running >5x the token throughput per thread can’t be overstated. Add in the multi-threading advantage and it’s a knock-out punch for local LLMs.

Local inference has a role: if you’re working with extremely private matters or you want an uncapped model that will talk dirty or generate NSFW photos, local is the only option. I think Apple and others will continue to also run a lot of useful workloads locally such as text editing suggestions, speech to text, text to speech, and image manipulation. As local hardware improves, these capabilities will get better too.

But, for most LLM work, the cloud will continue to dominate for a long time to come, if not forever.

[yieldcrv]

I think local inference will be fast enough

There is so much happening in that scene, where tokens/sec double or 10x

So I could see the same hardware doing 20 tokens/sec on a large model suddenly doing 200 tokens/sec in the future, a better device in the future doing 500 tokens/sec, while having vision models baked in, audio models etc

Users wont consciously switch to local, they will just have it and use it

[toofy]

and a toyota camry can’t compete with a lamborghini but the camry vastly outsells them.

you’re far far far more likely to see a camry or equivalent in an americans driveway than you are a super car.

you’re also more likely to see an enthusiast with a corvette or equivalent muscle car spending way more than it’s worth to tinker on that car in their garage than you are a super car.

[anon373839]

> A Mac Mini can run one LLM inference thread at a time.

That’s not accurate. With MLX, at least, parallel inference is both possible and useful. Model serving tools like LM Studio and oMLX support parallel generation with continuous batching, and the total throughput increases with it.

[ttul]

You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be able to compete cost effectively in your home lab with a cloud that has >100M end users streaming millions of inference requests per second across a gigantic fleet of machines.

Can I run a few inferences in parallel on my Mac Mini? Yes. But put 1,000 Mac Minis in a datacenter serving 1,000 copies of myself? That's going to be more efficient.

[anon373839]

Oh certainly, a local machine can't parallelize like the cloud can. Does anyone think that, though? What it can do is provide good-enough inference to support common business needs, without creating unnecessary data risks.

[dickersnoodle]

Why on earth would you need to do that for most things? Not everyone wants to be the next Coreweave. It sounds like you're killing a fly with a cannon to me.

[subarctic]

> Most useful LLM work is done in parallel

I guess what I'm doing is not considered that useful then? I usually only have zero, one, or occasionally two things actively doing inference at a time, be it claude code sessions or one of the chatgpt/claude web interfaces, and i bet that's true for like 95% of people using llms. And anyway i bet even the hardcore people using a bunch of parallel agents would appreciate having access to local, private inference for some things.

You're obviously right though that cloud inference isn't going away anytime soon

[jghn]

What about when claude code starts spawning sub-agents? it's pretty good about doing that now even if a user doesn't request it. Does this could as parallel inference?

[subarctic]

I haven't seen it do it by itself that much so far. Even then, a lot of the time is spent waiting for some shell command to complete, which isn't inference. I guess being able to do parallel inference is like having multiple cores in a cpu, you can get by without it by just sharing that single core, and having it just gives you a performance boost under certain circumstances

[jghn]

Interesting. For me it seems to be much more common that even basic requests start spinning up sub-agents. But I also routinely ask it to do this so maybe it's stored this as a preference somewhere deep in its bowels.

And yes, I agree. I find the experience better less for speed and more for context management. But it's far from necessary.

[master_crab]

100%. Orchestrating multiple sub agents is a key requirement for many workflows. If only because it minimizes context sprawl (amongst multiple other reasons).

There is probably a middle ground though. Something like a main thread agent (say a local model for Hermes) orchestrating cloud based sub agents.

[varispeed]

You can always buy multiple Macs. I think Apple's great differentiator could be making frontier class models local. I you want more threads, buy more Macs.

I don't want to run any workflows on someone else's computers.

[7speter]

As far as I know, there isn’t an interface like nvlink that allows these macs to work in tandem; they would just send their data over ethernet, maybe thunderbolt/usb-c?

[fragmede]

Why yes you can! https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-stu...

[wolvoleo]

> 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally

Well just because they don't sell them. Doesn't mean that will be the case forever.

[Danox]

You lost it at when you said forever…