I don’t think we should use current prices as landmarks for large scale demand. That Studio’s current prices is inflated because of a (presumably) short term supply crunch, not because the average user is willing to pay $24k for a home AI inference device.
It assumes that RAM remains supply constrained and that none of the existing RAM contracts are cut short.
But Meta and xAI putting A TON of AI compute onto the market. OpenAI and Anthropic are raising the costs of inference (by reducing how much inference users get via subscriptions). And we haven’t seen Oracle / CoreWeave struggle to pay their debts yet, but they will be selling assets once they get close to that point.
Funnily the current high end Mac Studio are not suited for current LLMs. M3 Ultra is "quite an old" chip for AI, despite its bandwidth. The issue for running local models (especially LLMs), you need few things to align really well: 1) compute power (affecting PP) 2) VRAM capacity (affecting model size you can load) 3) Bandwidth (somewhat affecting decoding speed).
The issue with the M3 chip is the compute performance, as it doesn't fit well the transformer architecture. This changed with the M5 (apple baked their own matmul into the chip), which would significantly speed up PP (and video/image generation btw), making the M5 Ultra significantly faster than the M3 Ultra and in practice much more usable. You can try to load Kimi or GLM on M3 Ultra, but it's not usable. Now the M5 Ultra is not out yet, but undoubtedly it will be a superior offering, and shilling 15k on 512GB version is actually reasonable (if it's every priced remotely around that tag).
By the way, they cut the supply because they run out of RAM. They still sell the 96 GB model. Everything else is gone, no stock, they cannot manufacture them. Even minis are 48GB max!
If demand doesn't fall down or current manufacturers supply go up, somebody (presumably in China) will spin up fabs. Apple wanted to use blacklisted Chinese RAM already.
DDR5 is still mostly made with DUV (remember Intel 14+++++++++?), and even though manufacturers have slowly been moving a few layers to EUV the advantage is at the margin. Lack of EUV at scale will not prevent China from ramping useful RAM into this market.
Chinese fabs might not be so tied with red tape and regulation upon regulation (which is a funny reversal, in terms of "communism vs capitalism" bureucracy/inefficiency cold war thinking)
Ebay and Amazon are flooded with it. Especially if you are looking for anything prior to DDR5. DDR2 and DDR3 are especially flooded with weird brands you've never heard of before.
Unfortunately its not so cheap anymore as everyone ramped prices up of course.
Last year I could still get 32GB of DDR4 for under $60 from chinese brands.
If the increased demand is not short term, production capacity will eventually increase. In the meantime, the logistics disruptions and industrial material shortages and energy inflation will disappear as soon as the wars disrupting them stop, which should bring prices down.
If demand and prices keep rising without production capacity being built fast enough, there will likely eventually be a rush leading to overinvestment and price crashes, but there are too many other factors involved; state investment for security, international politics and trade relations, the possibility of an AI bubble burst, etc.
There are wars coming. The prices are not going down.
We are in a bubble which will be burst the moment the world starts retaliating against the US' 20+ year history of supporting genocide and committing war crimes unabated.
The underlying demand for massive RAM builds is coming from AI hyperscalers.
There are lots of signals that the sector has been overinvested and that corporate customers are pulling back on spending as the cost of the APIs is revealed.
Once the hyperscalers start struggling to bay their debts (it will happen, just a question of time), there will be a supply glut.
So the only question is: do we share the same definition of “short term”.
Unless the raw materials have an inherent limit on mining/production due to the amount present on the planet, why should or would companies not ramp up to eventually meet demand?
Edit: Okay, this doesn’t mean that that’s actually possible in the short-term, so I think you’re right. But that means as the silver lining, in the medium term horizon there’ll be enough supply again? :’)
For existing producers expanding capacity would be a risky move. But it's the perfect time for any newcomers to enter the market. Low yields and worse product don't matter as much right now, and by the time the market cools down you have everything dialed in and can compete on even ground
> it's the perfect time for any newcomers to enter the market
This is a good hypothesis. Curious if anyone has data on the failure rates of new entrants in semiconductors based on how frothy it was on founding.
On one hand, more demand makes selling easier. On the other hand, a shortage makes your input costs (consumable and capital) pricier.
EDIT: It seems like the 2 to 3 year lead time and a crowding effect from new entrants historically made booting up a fab into a boom a bad bet [1]. (The article argues, convincingly, that this time may be different.)
I heard that China was spinning up DDR5 (but not HBM?) production in the next couple of years, with the hope of outcompeting Korea and Taiwan in the mid to long term.
Thanks for the link (and underlying thoughts), I really hadn’t considered that.
So essentially, due to technological progress and other factors inducing price collapses (or at least cycles), you can’t start stockpiling insane amounts of finished-product semiconductor, which means you can’t scale production at current technology levels to infinity either?
In 2010 one of the standard configurations for the Mac Pro was $4,999. Once you customised ram, storage, peripherals and software it could easily end up above $15,000, or $23k today accounting for inflation. Apple hardware is one thing that has actually got cheaper over time.
Or they could use that same amount of memory to ship 64x Macbook Neos, and probably make higher margins off the hardware volume.
Those Macbook Neo users would be very reliant on Apple intelligence, enough maybe to pay for a service with it. I think Apple's much happier going this path.
> Or they could use that same amount of memory to ship 64x Macbook Neos, and probably make higher margins off the hardware volume
If it's an "or," absolutely. But if it's an or, they should be prioritising Macbooks over the Mac Mini Doug Brooks is discussing.
When we breach the "and" of memory supply sufficient to allow for more Mac minis (and Mac Studios), I think it would make sense to consider relaunching Xserve (with new branding, of course) as a consumer/small business product.
Memory supply isn't what held back XServe. We wouldn't need XServe if Apple treated the Mac like a regular computer and supported usable, first-class headless workflows and eGPUs.
The writing has been on the wall since 2019. Apple doesn't like the old way of computing, their goal is to expand the ecosystem by prioritizing install-base and then pushing first-party service offerings like they did with the iPhone. And like they did with the iPhone, Apple is great at ignoring power users to focus on features that make them more money.
You may be waiting a few decades for this type of product, memory supply be damned.
It's really not. Apple's phone margins have been as high as 30-40% per-unit, it's likely that they make at least ~$80-150 per Macbook Neo sold.
At the $150 mark (which is probably accurate factoring in lifetime service spend), that's a $10,000 minimum return on the 64x Macbook Neos. Apple can charge that type of premium on consumer hardware, but they're in no position to command $10,000 margins on professional hardware. They're not Nvidia, Apple has always been LARPing as an HPC vendor.
But surely you understand how your preferred customer is the less profitable one?
Apple won't subsidize these low-margin enthusiast products with the profits made from services and higher-margin hardware. Tim Cook would much rather ship the 64 Macs, and get ~15-20 school-age kids hooked on Apple One or the App Store for the rest of their life. There's understandably not much patience for catering to people that want to opt-out of the Apple Intelligence service ecosystem, effectively leeching off of more successful products. The volume and opportunity cost kills the concept in the cradle.
Apple sure doesn't act like it. The Mac is still a minority market share of PCs, and their entrants into spaces like AR do nothing to compete with incumbents.
Now that the Mac Pro is depreciated, Apple's plan to pivot to service offerings seems set in stone. That's the "want it all" attitude they've adopted with the App Store.
I think so, too, and I think it'll end up being a race between Apple & NVIDIA (or NVIDIA partners) to see who realizes this first. It would probably be easier for Apple to do it because it wouldn't require a form factor adjustment [over the Mac Studio they already have]. That said, NVIDIA already offers chipsets for both the lower end (DGX Spark with Vera + GB10, at roughly the $4500 price point) and higher end (DGX Station with Vera + GB300, for $85-100k). The DGX Station is equivalent to ~5-6 RTX6000 GPUs attached to a mid-range CPU server, but far more than most individual developers would want or need. I've heard through the grapevine that NVIDIA's received consistent feedback that they need something like a "GB20" that slots above the Spark/GB10 and can simultaneously run larger models for inference while hosting a dev environment on the same box. You can daisy-chain Sparks just like you can daisy-chain Mac Minis, but you're still constrained on model performance based on what a single device can accommodate.
Form factor is the easy part - both Nvidia and Apple are experienced SOC designers.
The hard part is the GPU architecture. Apple Silicon was designed with a laser focus on raster efficiency (similar to AMD's GPUs) which makes a lot of sense for highly mobile hardware, but is a crippling mistake for high-performance compute. Apple's largest Ultra chips are hamstrung with SOC-tier GPU performance, their highest-end desktops are outperformed by Nvidia's laptop offerings. Apple has to find a way to scale upwards without imposing too much architectural strain on their cheaper hardware like the iPhone and Macbook. Nvidia has already solved this issue; full CUDA compute stacks are usable on extremely cheap GPUs like the Nintendo Switch's Tegra SOC, or the Mac Mini-sized Jetson boards.
In terms of "who needs to redesign more to address the market", Apple has a lot of technical debt to unearth before they catch up to Nvidia. And if they do catch up, Nvidia will still support Linux and other differentiating features that Apple refuses to implement. It definitely feels like Nvidia is closer to a winner with the Spark than Apple is with the Mini or Studio.
It assumes that RAM remains supply constrained and that none of the existing RAM contracts are cut short.
But Meta and xAI putting A TON of AI compute onto the market. OpenAI and Anthropic are raising the costs of inference (by reducing how much inference users get via subscriptions). And we haven’t seen Oracle / CoreWeave struggle to pay their debts yet, but they will be selling assets once they get close to that point.