I make a point of never going beyond about 220k, unless absolutely necessary (and it's almost never necessary), anyway, even with models that degrade more slowly, so this is just a discount.
Heck, agents don't start editing before they're already at 70k for me.
I've played with explorer agents giving exploration summaries to help the implementer agents use more of their context for implementation, but it doesn't work as well. There's always something lost in the handoff.
My understanding is that models can suffer “context limit anxiety” and a higher context limit will make it perform better especially as the context expands
This seems functionally similar to OpenAI having a step in pricing once you exceed a certain context length (also at 272k aka 2^18 aka 256k).
Having a lot of active context increases the per-token cost (flops issued and bytes read per token out) so it makes sense to pass that cost on to users. I'm actually surprised it's implemented as a hard cutoff instead of a smooth gradient.
You only need to cache for tokens that are actually being used. Using 100k tokens of a 256k token window takes the amount of memory as using 100k tokens of a 1m token window.
which is why they charge extra for an option to hold >256K tokens.
The cost may be smoothly variable, but likely there's a bimodal distribution of users who barely use any context and users who push it to the max. Average price across both extremes fits nobody, but averages per kind of workload can be close enough.
Having it as a separate model makes it easier to load-balance the traffic.
No, the main issue is that it's hard to communicate pricing where token price increases as token count increases, and they figure they can approximate the parabola well enough with two lines.
If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.
And my guess is there's probably not a huge amount of customers that want somewhere in between: if you're willing to pay the long context surcharge; you're probably semi-price-insensitive anyway to just use 1M.
> If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.
That seems like it is not an awfully hard problem, because the AI buildout scale is so massive. You simply don't provision too many of the smaller buckets, and when they are fully loaded you route the traffic to the larger buckets. Now you don't have a utilisation problem.
Now of course you can simply organize it that way internally, and present a single price to the customer. Perhaps thats the smarter thing to do.
I don't think so. This is a separate model, so I assume that if you just use this and switch to the 1 million context model when you reach 256k, your cache will be invalidated, so you'll re-pay the 256k tokens on the 1 million context model pricing.
Edit: I was wrong, thanks to longwave for pointing this out. It's absolutely possible to start out on the 256k model and then switch to the 1 million model when you get close to the context limit without invalidating the cache:
"When switching from k3-256k to k3 (1M), if k3-256k is close to the 256k limit and you don't want compact to lose information, you can switch directly to 1M. The current version switching from 256k to 1M does not affect the cache."
Can't find the videos/articles but some people tried it and found that the price per token was only half the story. It seems that it uses a lot more token, coming back to similar prices with other models.
I was so excited that it is open source until i realised the model required 1.5To of VRAM. Unsloth has compressed it in 1bit at about 570gb VRAM with 75% accuracy, that's almost mac studio territory...
As I understand it, they would have to train a whole knew model to hard cap it's context to different lengths. That would be cheaper to train and had cheaper inf, but still a huge investment.
Kimi K3 supports a context window of up to 1 million tokens. We achieve this through extending
the context window progressively as training proceeds, following a four-stage curriculum. The
window grows from 8K to 64K tokens during pre-training, and from 256K to 1M tokens during the
cooldown phase.
You absolutely don't need to 'retrain' to reduce your context window. In vLLM it is an inference parameter. Smaller context window, smaller KV, less RAM needed to serve the same volume of requests.
With a lot of architectures, you technically don't need to retrain to extend the context window either; e.g. RoPE scaling; but performance is typically crap.
When I first built features with GLM, there were lots of bugs, and it took me ages to fix them manually. Now the features implemented with GLM 2.0 have almost no critical bugs after testing. I can’t even imagine how capable K3 will be. It may well be on a level that ordinary people cannot access.
It's pretty handy for have very long contexts for long running agents, or else when doing literary analysis to simply be able to load the entire book in.
Using Claude/ChatGPT I rarely hit >256k context for most basic coding tasks. Sometimes if I need to do something more "intense" (bigger refactors, new features) the 1mil window is nice.
When working on hard problems (not "vibecode me a script to show an alert box", but e.g. "let's see what this three-level LUT-state-machine obfuscated binary does"), hitting the 1M (!) context window with Claude Code feels like you were talking to Claude Claudewski when his shift just abruptly ends, he packs his things, throws the office keys at Claude Claudeson in-between the front door frame while handovering like "Hi! Nice to see you, good luck." and now here we go again, you are working with someone who just experienced an acute amnesia. It tries everything it already tried, everything it was told in the initial prompt to not do, everything it was told in follow-up prompts not to do. "You were right, this approach does not work and we don't have 20 TB RAM on this machine for full symbolic execution, let me try..."
In Codex, it's so seamless that I sometimes just notice "wait, the context was 20 % remaining, it is 70 % now, wow, when did this happen", while it seamlessly works on the task. Basically never had an issue with context on Codex, be it coding features, cracking hard crack-me ciphers, or researching basically anything.
(For full disclosure, my last experience with Claude was a few weeks ago when I cancelled the subscription, maybe they fully reworked the traumatic "Summarizing" - "Oh, hi! Where are we? Who am I? What we are doing? This is taking too long, let me take a shortcut..." lobotomy they were doing in the meantime.)
256k is enough when the harness uses it properly and the model is not stupid. And also when the tokenizer is not tuned to invoice as many tokens as possible...
I'll have to give it a shot some time soon. I just can't imagine working on any of my serious projects doing a new large feature with that little room. By the time it has looked at half the code required to start planning the work, it'd be out of context. Must use sub agents better or something.
Codex turns over 256k windows for *days/weeks on end* without seemingly much fidelity loss, especially with a goal that anchors it to a durable plan/ledger.
Since Claude is the first time for me really, really out (TIL against my wished about https://status.claude.com/), I am now interested enough to see what else works. But ... when I click pricing, I see "Join a waitlist". Wtf? Are they really that good, so were totally surprised and overwhelmed by the requests, is this a marketing stunt, or do they just don't have the hardware being in china?
As a user, I much prefer that to service disruptions or severely degraded or secretly quantized performance. However if I didn't have an account, I'd be pretty pissed off about not being able to give them money and become a user.
Kimi had become that popular. I was a subscriber of Kimi back when latest version was Kimi K2. Later I unsubscribed because I jumped over to GLM subscription (they had amazing deal). Now when I wanted to try out Kimi K3 to find out what the fuzz was all about, I couldn’t subscribe to them.
I remember reading a post from Moonshot team about this, they are doing this because they are almost at peak capacity and want to reserve it to keep the quality for their current customers.
We are actually witnessing an open-weight model catching up at catching mainstream users attention. And instead of behaving like Anthropic, they actually care about their users experience.
The only downside with third party providers is that you have to trust that the provider have setup and configured it correctly, and is not secretly quantizing it.
Yes, something I read just before it was released suggested that, because of the unique features of the model, there was a back and forth of Hugging Face, Moonshot and providers like Together AI and Fireworks AI. This also explained why it took much less than a day for Together, Fireworks etc to appear. Whatever they are doing is what Moonshot wants, I think.
Not at all a marketing gimmick, the demand is simply that high.
I was able to press "Join waitlist" and then within 48 hours got accepted. The limits aren't very high, no where near the endless subsided+resets given on ChatGPT/Claude. I recommend ChatGPT for good value output!
Others here mentioned "the providers could be quantizing it!" but some of the providers on OpenRouter have partnered with Moonshoot and OpenRouter shows the int when you expand on the provider.
It should say "mxfp4" but providers like Baseten report FP8.
It's an extremely popular model hosted by a company affected by hardware export bans. I doubt they'd voluntarily prevent people from subscribing if they didn't absolutely have to to maintain service quality.
I honestly prefer this as opposed to what Anthropic has done in the past which is to continue accepting users even if they don't have the capacity to serve and behind the scenes tone down everyone's limits.
tintinweb/pi-subagents. Then I use superpowers and it spins up subagents automatically. Works really well and you can customise the models and efforts you want to use with local AGENTS.md.
I’m primarily using gpt-5.6 and then opus for reviews.
Their harness is indeed nice, but we're using it against our own AI Gateway at work. Right now we only expose GPT models on it, but I imagine it's possible to add our own models eventually
Open code is cool once I added the ctrl-o function to it (show thinking and command outputs at will instead of on by default), but sadly I don't think it got merged by the team.
...but we're talking about compaction, and opencode's compaction is (or was) terrible. I've seen so many horrible problems that I keep it disabled (with an envvar flag, because even the config flag to turn it off was broken).
I have so much gratitude to the frontier companies who did all the extremely complicated research and development, model by model. It already feels difficult to remember how much capital it really took. Thank you for getting us to this point.
At least in the self-hosted LLM inference engines, you have to pre-allocate space for the maximum amount of context you want to allow for each parallel session. By using a lower maximum, you don't have to allocate as much VRAM for each session, allowing more usage for the same amount of hardware. Thus, cheaper.
It seems to only be available in Kimi Code, via subscription, no there's no API pricing. The linked page says it consumes about half as much quota as the 1M version though.
Not relevant to this link but I was thinking about the allegations of Chinese AI companies distilling from the big frontier American ones. And I came to the conclusion: I don’t care.
Who cares? China has always copied and then copied the means of production and then out produced. See also Tesla and now all the Chinese cars eating their lunch.
As long as I get really solid AI models for cheap that do what I need I don’t care if they’re Chinese or otherwise.
I’ll still never use Grok from SpaceX AI cuz eww no, I have principles. ;-)
It feels like we're moving away from "bigger context is always better" toward "right-sized context". I love this not just because my wallet feels safer but because most of my coding sessions never come close to needing 1M tokens anyway
A bit of topic. But how likely is it that the US will restrict Chinese open weight models and also force Euro countries to do the same? I think it will be effective within 6 months. The US is having a hard time staying competitive.
It isn't legally possible for them to do this at the EU level. The EU parliament would never vote for it.
For pressure at the country level leading to this kind of thing I think it's very unlikely. Here in Sweden it wouldn't just require a vote in the Swedish parliament and before this there'd have to be förarbeten and you can't just brazenly push things through with insane arguments, Swedish social convention goes against it-- and there's just no way to get it through.
It also might not even be legal. "We aren't at war with China and I'm a communist, and the US LLMs are so aligned with values inimical to my political ideology that this is interference with opinion formation" might be an actual legal argument that the ECHR or CJEU might actually have to accept.
Well, that's the deal, I assume-- that they weren't allowed to buy the Japanese light source outright, so they bought the American one, even though it required giving the Americans some sort of veto or control.
I guess it sucks if one wants to expert broadly, but if you're big on vertical integration and the Japanese won't sell I guess you take what you can get.
Hopefully this helps reduce some of the pressure on their infrastructure. Their models have all become super dumb recently and their support are not addressing it. I have a hunch they’ve been serving a significant percentage of requests with quantised models.
Why are Anthropic and OpenAI even allowing their coding harness apps to be plugged into different model providers…? I’m surprised they haven’t figured out a way to clamp down on that by now.
From my view, as soon as they do that, they send people out the door to use Opencode instead - and once many people have a taste of trying every model via Openrouter, it's eye opening as to the possibilities.
Of course - Anthropic and OpenAI have an advantage in the amount they can subsidize the usage, but I think those days are waning.
Well when a huge part of potential revenue is all in on Bedrock... you need the harness to be able to talk to Bedrock. And Vertex. And all the other places these models are hosted. And allow for proxy because many businesses do not all direct internet access... all valid business reasons.