[habosa]

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat.

Just writing this down so I can be praised/mocked in 5 years.

[_verandaguy]

I'm perfectly happy with Apple not becoming an "everything we do is AI-centric" business.

I'm fatigued by it all at this point. It's streamlining the interesting and fun parts out of my job (by practical necessity of use there), and if I used it half as much outside of work I'm sure it'd do the same there too.

[operatingthetan]

>I'm fatigued by it all at this point.

This is the prevailing opinion of people even outside of tech.

[_verandaguy]

I know public opinion polling supports that, but the parts of my social circle which are outside of tech seem to be, at worst, apathetic (and at best enthusiastic, though that's not a big fraction).

That said, I think it's a good thing that this sentiment is coming to the forefront.

[nsagent]

I was actually surprised to hear my brother-in-law deride LLMs as being useless for areas he has expertise in when I visited for the 4th of July.

He was complaining that he would ask how to perform a certain repair on a car, and the LLMs he tried (ChatGPT & Grok) would give him a long involved process and he'd ask why not do it this simpler way and it would say, oh you're right! He just found it gave bad advice and realized (rightly) that in areas he has less expertise in he has no way to judge how good the outputs are.

This is from a guy who loves tech, historically worshipped Elon, loves his Tesla, and (rightfully again) didn't buy into SpaceX because he thought it was overvalued.

In the past when I visited for holidays he was liable to have a positive outlook on LLMs and their utility. Seems telling that he's starting to see the cracks.

[doikor]

> it would say, oh you're right!

This is easily the biggest problem with the current models. The models are just way too eager to please / say yes to the point that the models are happy to lie/make shit up if it means it can say yes.

[operatingthetan]

Yep. In my areas of expertise I can easily catch it coming up with wrong information, bad calculations etc. So laypeople are probably being led astray quite often.

[dehrmann]

There's a lot of Gell Mann amnesia going on with LLMs.

[amazingman]

> It's streamlining the interesting and fun parts out of my job

Interesting. For me it's streamlining the tedious and attentionally taxing parts of my work tasks. I love solving problems, I don't particularly love shaving yaks.

[QuercusMax]

If i'm gonna shave a yak, I'd prefer to do it with a set of powered clippers, not a pair of embroidery scissors.

[ttul]

Here’s the two main reasons why local inference won’t compete any time soon with the cloud:

1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware.

2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advantages of running >5x the token throughput per thread can’t be overstated. Add in the multi-threading advantage and it’s a knock-out punch for local LLMs.

Local inference has a role: if you’re working with extremely private matters or you want an uncapped model that will talk dirty or generate NSFW photos, local is the only option. I think Apple and others will continue to also run a lot of useful workloads locally such as text editing suggestions, speech to text, text to speech, and image manipulation. As local hardware improves, these capabilities will get better too.

But, for most LLM work, the cloud will continue to dominate for a long time to come, if not forever.

[yieldcrv]

I think local inference will be fast enough

There is so much happening in that scene, where tokens/sec double or 10x

So I could see the same hardware doing 20 tokens/sec on a large model suddenly doing 200 tokens/sec in the future, a better device in the future doing 500 tokens/sec, while having vision models baked in, audio models etc

Users wont consciously switch to local, they will just have it and use it

[toofy]

and a toyota camry can’t compete with a lamborghini but the camry vastly outsells them.

you’re far far far more likely to see a camry or equivalent in an americans driveway than you are a super car.

you’re also more likely to see an enthusiast with a corvette or equivalent muscle car spending way more than it’s worth to tinker on that car in their garage than you are a super car.

[anon373839]

> A Mac Mini can run one LLM inference thread at a time.

That’s not accurate. With MLX, at least, parallel inference is both possible and useful. Model serving tools like LM Studio and oMLX support parallel generation with continuous batching, and the total throughput increases with it.

[ttul]

You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be able to compete cost effectively in your home lab with a cloud that has >100M end users streaming millions of inference requests per second across a gigantic fleet of machines.

Can I run a few inferences in parallel on my Mac Mini? Yes. But put 1,000 Mac Minis in a datacenter serving 1,000 copies of myself? That's going to be more efficient.

[anon373839]

Oh certainly, a local machine can't parallelize like the cloud can. Does anyone think that, though? What it can do is provide good-enough inference to support common business needs, without creating unnecessary data risks.

[dickersnoodle]

Why on earth would you need to do that for most things? Not everyone wants to be the next Coreweave. It sounds like you're killing a fly with a cannon to me.

[subarctic]

> Most useful LLM work is done in parallel

I guess what I'm doing is not considered that useful then? I usually only have zero, one, or occasionally two things actively doing inference at a time, be it claude code sessions or one of the chatgpt/claude web interfaces, and i bet that's true for like 95% of people using llms. And anyway i bet even the hardcore people using a bunch of parallel agents would appreciate having access to local, private inference for some things.

You're obviously right though that cloud inference isn't going away anytime soon

[jghn]

What about when claude code starts spawning sub-agents? it's pretty good about doing that now even if a user doesn't request it. Does this could as parallel inference?

[subarctic]

I haven't seen it do it by itself that much so far. Even then, a lot of the time is spent waiting for some shell command to complete, which isn't inference. I guess being able to do parallel inference is like having multiple cores in a cpu, you can get by without it by just sharing that single core, and having it just gives you a performance boost under certain circumstances

[jghn]

Interesting. For me it seems to be much more common that even basic requests start spinning up sub-agents. But I also routinely ask it to do this so maybe it's stored this as a preference somewhere deep in its bowels.

And yes, I agree. I find the experience better less for speed and more for context management. But it's far from necessary.

[master_crab]

100%. Orchestrating multiple sub agents is a key requirement for many workflows. If only because it minimizes context sprawl (amongst multiple other reasons).

There is probably a middle ground though. Something like a main thread agent (say a local model for Hermes) orchestrating cloud based sub agents.

[varispeed]

You can always buy multiple Macs. I think Apple's great differentiator could be making frontier class models local. I you want more threads, buy more Macs.

I don't want to run any workflows on someone else's computers.

[7speter]

As far as I know, there isn’t an interface like nvlink that allows these macs to work in tandem; they would just send their data over ethernet, maybe thunderbolt/usb-c?

[fragmede]

Why yes you can! https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-stu...

[wolvoleo]

> 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally

Well just because they don't sell them. Doesn't mean that will be the case forever.

[Danox]

You lost it at when you said forever…

[ceroxylon]

I don't mind the subtle ML integrations that they have put in the photos app: plant ID, recognizing faces, removing background, OCR text search (even for handwriting!), etc.

[eastbound]

You’re listing every feature I’d like to remove; Meanwhile Android users have great holiday pictures at the bottom of the Pyramids because they can remove the people from their pictures.

[rdn]

I don't like when I'm trying to copy a photo online and it copies some piece of text in the photo instead

[dabbz]

I suspect we'll see a hybrid before an all or nothing. Local models for computer control or delegating, online models for things that need strong reasoning, planning, and knowledge access. Again, I'd be more than happy to be wrong. I just see models growing faster than the hardware can.

[QuercusMax]

I'm also of this opinion, but also that it doesn't have to be Apple (but they are well positioned). What I've seen with running local models on my 48GB M4 MBP is really impressive - it's not the same level as hosted stuff, but it's better than what I was using a year or two ago.

[ingenieroariel]

I wrote: "we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.)" April, 2023

We are both late and early.

https://news.ycombinator.com/item?id=35527692

[bigyabai]

You should not buy a fully loaded Mac Studio for AI unless you absolutely NEED macOS. You will be wasting so much electricity idling on prefill while your GPU pulls 150-250w from the wall.

Buy an Nvidia Spark, then whatever cheap Mac you want to use as a thin client. There's no reason to force Apple Silicon's round peg into a square hole like AI inference.

[timy2shoes]

Benchmarking that I've seen shows that the M5 Max outperforms the DGX Spark, e.g. https://www.reddit.com/r/LocalLLaMA/comments/1tfzsd6/m5_vs_d... or https://www.reddit.com/r/LocalLLaMA/comments/1tr7hzw/psa/. Seems to me that Apple is doing pretty well with local AI inference.

[bigyabai]

Outperforms doing what? Inference is not a homogeneous workload, memory bandwidth correlates to decode speed and layer swapping but not necessarily inference speed overall.

The other half of that equation is latency, predicated on prefill performance which needs a powerful GPU and ideally ALU-level optimization to build larger KV caches quickly. Even the M5 gets smoked in this department, the M5 Max has a 50% longer TTFT on Qwen's 27b dense model at only 16k of context, which is a pretty typical starting context to use for agentic editing in normal apps like OpenCode/Claude Code: https://raw.githubusercontent.com/Osmantic/MMBT-Messy-Model-...

For agentic, 50-256k token on-device coding sessions, the Spark will be faster and consume less power running larger models. Without an external GPU (which Apple doesn't support), Apple Silicon will always be bottlenecked during prefill. Apple's failure to address this with their GPU architecture is a big reason why Apple Silicon viewed as a waste of time and money for professional datacenter deployment.

[mercutio2]

For agentic work, you just cache the prefill kv cache of the relevant system prompts. TTFT is a little slow (10s of seconds, oh no!) the first time you boot up a new harness.

I keep hearing people make this claim that TTFT is a problem, and… it just isn’t, if you’re running oMLX.

Folks in my camp keep saying this, and folks in your camp keep beating a drum we tell you isn’t resonating. Not sure why I keep bothering to argue; you can’t buy a high RAM Mac Studio like mine anymore.

[bigyabai]

It's really not that hard. If Apple Silicon was competitive for inference, then Apple would not be shelving them: https://9to5mac.com/2026/03/02/some-apple-ai-servers-are-rep...

They're not usable for deployment. They're perfectly fine for "enthusiast" low-end usage with 10-30B models, but the same goes for almost every dGPU made in the last 10 years. Your Mac Studio cannot run frontier LLMs at an interactive speed, even Apple has given up on using it as an inference backbone.

[j45]

I'm not sure that's what this article is about.

Apple is doing something very different. Their AI experience for end users definitely has been a little behind.

Apple Silicon, however, has been quite unique for the last 4-6 years and it's increasing overlap with LLMS.

The model/chip optimizations are definitely improvements, the thing that is really standing out the past 2 years is how much the open source model community has been making possible, especially when you know a group of use cases.

[rk06]

There is common wisdom: during gold rush, sell shovels.

Apple's shovel (ahem, Mac mini) is the highest quality.with Companies burning money left, right and center, Apple can dispense with advertising altogether

[xyst]

Apple has failed to live up to the Steve Jobs era and initial iPhone hype. It's not just "AI experiences", it's computing in general. Maybe the consumer sector is just dead/dying. Maybe consumers are just running out of cash because filthy VCs are destroying communities and forcing the 99% into poverty.

The one thing that is marginally exciting: the Apple SoC or M series chips.

It's unfortunate they are locked behind crappy macOS and other proprietary apple crap.

[overfeed]

> it's computing in general.

Unsurprising. Apple seriously thought the iPad would replace computers and usher in a "post-PC" word during their "What is a computer?" ad campaign era. Now they are sticking phone chips in laptop chassis.

[mlsu]

For the general consumer, they were basically right though. Most people don't use laptops except for work. The primary computing device is the phone, and phones have basically become become mini-ipads in form factor since that ad aired.

[dwaite]

> Apple seriously thought the iPad would replace computers

...for some users. See their "Mac is a truck" analogy.

And it has. My parents haven't owned a Windows or Mac machine in six years, since they got rid of the one I gifted to them a decade ago. Its all iPad and iPhone.

[giancarlostoro]

Its a shame RAM prices are getting in the way of Apple. The 512 GB RAM Mac Studios would have been worthwhile.

[ransom1538]

"Apple has totally failed to deliver interesting AI experiences so far ..."

You think this is a mistake...

[nvme0n1p1]

https://www.crikey.com.au/2025/01/08/apple-new-artificial-in...

Of course. Do you think this was on purpose? All part of Apple's brilliant master plan?

[thesurlydev]

Did you mean 5 months? :)