[datadrivenangel]

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model."

Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able to run models locally though.

[LeBit]

I’m an idiot who is unable to project itself in situations I’ve never experienced before.

So, I always thought local LLMs were toys not worth pursuing.

Only once have I tried something decent like Gemma 4 31B and Qwen 3.6 27B did I realize how incredibly useful they are.

You stop fearing you are sharing sensitive information.

You stop fearing you will run out of tokens.

You stop fearing about the availability of the remote AI.

Local LLMs are extremely valuable.

[bityard]

*for many tasks

[mips_avatar]

The cool thing about the 3090s is the RAM bandwidth. Token generation is mostly bottlenecked on memory bandwidth. Dual 3090s have 1.87 TB/s memory bandwidth (0.936 TB/s each), vs the M5 Macbook pro with only 0.3 TB/s (max chip has up to 0.63 TB/s but it's a $10k machine at that config).

This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Also if you're running a big model on a macbook pro the UI gets laggy and the keyboard gets hot. Much better to run dual 3090s in your basement and connect to them from your Macbook.

[CobaltFire]

$4.8k for 48GB Max (what the parent said). Half of your quote.

Even a 128GB is $6.8k today. Still only 2/3 your quote.

Bandwidth is relevant (I have both a 5090 and an M4 Max 128GB Studio, so have direct comparison right here), but quote the cost appropriately!

[mips_avatar]

You need the 128gb ram config to get the 614 GB/s bandwidth (which is $6999), you could skip out on upgrading the storage to save money but at that point I think most people upgrade the storage too at which point it's $8-10k + tax.

[CobaltFire]

No? Any M5 Max with the upgraded GPU has the full bandwidth, which includes the 48GB model the original poster mentioned. Same as the M4 Max, where only the trimmed part had a lower bandwidth.

Why are you throwing in extra cost for something thats not necessary? I know multiple people with 128GB Macs and none of us upgraded the storage. Especially not on a Studio (which isn't currently available).

I will say that their $3k number is off. I somehow missed that, and its too low.

[mips_avatar]

I made a mistake, there is a $5k config with high memory bandwidth. The Max chip has two tiers (I incorrectly thought the tiers were based on memory capacity), you need the higher tier Max GPU upgrade (+$300) to get the 614 GB/s memory bandwidth but you don't need to upgrade the RAM to 128 GB to get the full memory bandwidth. So to get the 614 GB/s you need to upgrade to the max chip + upgraded GPU, but you can spec it at only 48gb if you want. So the total for an m5 max with 614 GB/s memory bandwidth is $4999-$9999 depending on config.

Still 3x lower memory Bandwidth than a dual 3090 setup which you can build for $3k with parts from facebook marketplace and run in your basement.

[titanomachy]

The bandwidth argument is compelling, do we have benchmarks for these models? I’m curious what it translates to in tokens per second

[mips_avatar]

I benchmarked mine for a deep research workload I was running. Concurrency 1 is the speed you'd get if you're chatting with one agent,

2x3090 (has an nvlink bridge though it didn't seem to matter hugely for inference)

Qwen 3.6 27b int4: Concurrency 1: 68 tok/s output Concurrency 32: 363 tok/s output Prompt processing speed: 1520 tok/s

Qwen 3.6 35ba3b int4: Concurrency 1: 150 tok/s output Concurrency 32: 1083 tok/s output Prompt processing speed: 4324 tok/s

Macbook Pro m3 36gb RAM: Qwen 3.6 27b int4: Concurrency 1: 18 tok/s output didn't measure the other metrics and it was a slightly different benchmark.

[titanomachy]

Yeah that’s a huge difference. I don’t think I want to interactively use any model with 18 tps.

[Aurornis]

I have an M5 MacBook Pro and I also have a separate GPU setup for running models. The difference in speed is significant. It's not just token generation speed, but time to first token (prompt processing).

The M5 hardware is amazing for what it is, but GPUs are still so much faster.

Running the models on the GPU box also means I can use the laptop on my lap instead of turning it into a hot plate.

[amelius]

What is your GPU setup?

[brcmthrowaway]

tinygpu kernel driver

[WithinReason]

I'm running Qwen3.6-27B on a single 24GB GPU at 80 tok/s, you don't even need 2 of them

[npodbielski]

Yeah but 4 bits very often loops needlessly. Which is not that bad because you do not pay for tokens. But you paid for hardware and you want use it for something useful. Q6 is better but then you have like 40t/s prefill. Which is really tiring. But at least it says sorry when you ask it what is wrong! I heard there is some extension for PI preventing that. I need to look into it. Otherwise I am quite happy.

[Zambyte]

"Very often" sounds like a lot more than I would say. I've been using Qwen 3.6 27b Q4 in Pi (with out any anti-looping extension) daily for weeks now, and I've had it get stuck in an infinite loop maybe 3 or 4 times.

[Der_Einzige]

You can fix looping with proper repetition penalties. Turn on the one called “DRY” that PeW invented and got merged into llama cpp

[npodbielski]

I added repetition penalty of two. I do not know. Maybe it is not applied correctly somehow by llama swap, that I am using, but I do not consider it that much of a nuisance to so I did not tried to fix it yet.

Do you have this DRY docs?

[jbellis]

That's a reasonable option, just be aware that you get about 1/3 as much memory bandwidth with the M5 Pro, or 2/3 with the M5 Max [now you're at $4100 for the lowest-end]. So both your prefill (flops-bound, M5 has a lot less) and decode (bw-bound) will be slower.

[amelius]

You can also buy a Jetson Orin with 64GB of unified memory.

[boredatoms]

The standalone mini/studio is better if you dont want to have a constantly hot laptop

Get a regular laptop and use the network to access the LLM

[satvikpendem]

To summarize a video I saw recently [0] rebutting your arguments, MacBooks can get crazy slow when running local models or even just Claude Code and Codex due to their poor implementation, to the point that the laptop itself becomes unusable.

There are other arguments for running an ssh-able box in a closet somewhere too as with KVMs you can give an agent remote control over the machine itself such that it has vastly more capabilities than if it were controlling its own machine it's running on, as well as not needing to keep the MacBook open all the time just to have the agent finish running.

[0] https://youtu.be/9tGrhrVKCrE