[zozbot234]

That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, around 27GB for a full 1Mtok context). What you're seeing in antirez's video is essentially the performance we should expect from a 128GiB system that has to load sparsely routed experts in full from disk because there's no real room for caching them.

192GiB Gorgon Halo systems will be an interesting future target for this model, the best you can do with 128GiB or less is probably to push batching higher in order to amortize the weights traffic over multiple inferences - which of course will sink single-session speeds even lower for a modest gain in total throughput.

[tito]

I read about its format being tough to compress further. What should we expect in terms of creating smaller models to run locally?

[whatsThisBtn4]

Would it even finish over a few weeks?

For AI agents this would take a year

[searealist]

No one will ever derive any utility from running models at this speed. Please prove me wrong. Give me the number of tokens input and output (and dont forget about reasoning) and acceptable time to wait for it and the use case.

[dannyw]

The OpenAI and Anthropic batch APIs give you a 50% discount.

This is like 98%.

[searealist]

I'm not aware of any service that gives you a 98% discount and is served off of M1s. Did you do the math for what this would cost vs K3 on openrouter?

[zozbot234]

antirez's own video (the one I referenced in my comment) shows K3 inference running on M5 Max, not M1-series silicon (which is OP). M1 series has far lower SSD read throughput and memory bandwidth, and can barely fit the model weights on its maxed out internal storage (2TB). (This is why the linked OP resorts to streaming the sparse parameters from the network which is incredibly slow.)

[searealist]

My answer doesn't change if it is M5s. Where is the math showing a 98% discount over K3 on openrouter. Heck, where is the math showing it is any % cheaper? How much electricity will your M5 sip to hit 1M input and 1M output tokens that would cost $3 + $15 there? I bet it is more expensive on the Mac.

[dannyw]

I just mean in terms of incremental inference cost.

If you already committed to your hardware, huge models at single or sub-digit TPS are still useful. LLM-as-judge is a good use case.

If your machine is gonna be idle overnight (and the power efficiency is excellent here), why pay openrouter if it’s not an interactive workload?

[searealist]

Because it's probably still more expensive in electricity costs compared to openrouter. Certainly it's not 98% cheaper.