What about 15k tokens per second? [0] I remember looking at this earlier in the year and it being so fast that it feels fake. And, yes, this model is old - but still awesome for what it is.
But it’s irrelevant. 750 tokens/s on a full frontier model is useful. 15000 poor quality tokens is much less useful no matter how much scaffolding you put around it.
You are missing the point. This is a technology demonstration on prototype hardware, and no one intends it to be seriously useful.
Their architecture has fundamental speed and efficiency advantages over GPUs or Cerebras. They expect to scale up to real LLMs by splitting a model layer-wise across several chips, which they can do without incurring any throughput penalty.
> They expect to scale up to real LLMs by splitting a model layer-wise across several chips, which they can do without incurring any throughput penalty.
I’ll patiently wait to see this in reality. Their demonstration hardware is a 250W chip that is enormous in die area for the model size. They’re making a lot of claims, but until they can deliver then it’s nearly vaporware in my view.
I’d be happy to be proven wrong, but I think they’re going to quickly run into hardware realities quite soon if they think they can just chain a bunch of chips together to achieve the same performance on larger sizes.
Actually it's the opposite. Per mm of silicon it's massively less efficient and making enough chips and powering them is a major bottleneck right now. Worse, scaling to larger models requires more of our absolute best quality silicon manufacturing, where e.g. an H200 mostly just needs more memory.
But I’m not missing the point. If you can run one frontier model at 750t/s, then you can probably run many many instances of an SLM in parallel at a rate that exceeds 15k/s. That’s kinda the point of the flash or ultrafast variants. And they’re on something much more modern than llama3.1.
Yeah, that's the point, right? With tool calling the LLM becomes code. So instead of asking it to write an accounting software, you can hire the LLM to be your accountant.
Not necessarily. Consider a human assistant who performs repetitive tasks at an acceptable cost and accuracy while dealing with edge cases often autonomously.
If we want reliability - we come up with processes to make it reliable and not rely on individuals getting it right. Code is a way to create a reliable process in the digital world.
For some things that's acceptable or even good. If I want to add up a list of a million numbers human assistants aren't bringing any advantages though.
I'd like to apply for this job where I can make mistakes and it's considered an acceptable cost. Seriously, I can't remember the last time I made a significant error and it was acceptable. Maybe during training? Half the effort in any job is literally verifying correctness.
Using gpt-5.4-mini in off-peak hours already feels like super-speed to me. That's probably no more than 100-150 tk/s. I can't imagine 750!
I've always eyed Cerebras but never had a use for it that would justify paying for the API directly. Although now that I think about it, trying out the API would probably cost less than a subscription for a month...
Agreed, 1000tok/s just fills up the context window (which is big by 2004 standards) super fast. But seems like 5.3-spark was just a taste of what’s to come.
The ChatGPT subscription gives you access to the -spark model(s) in Codex which are blazing fast (but pretty dumb) which I think runs on Cerebras hardware too.
is this specifically in codex? have been trying to use the models for months on opencode then pi but it says chatgpt subscriptions don't have access to it - i was under the assumption that OpenAI doesn't lock down their models based on harness a la Claude Code
Plus. No wonder - i suspected this but i couldnt find any docs. Side note, how are you liking Pro? I have really been considering getting Pro recently, but not sure if its more worth to just switch to openrouter. I feel like my usage currently barely outstrips the plus usage limits and Pro would be too much, and using Openrouter by default would mean I would have a lot more leeway to run more random lighter workloads without worrying about using up my limit, but I'll really miss GPT 5
I find it to be excellent. I have three Pro subscriptions so I can build stuff 24/7. I only use GPT 5.5 xhigh. Before 5.5 I wasted a lot of time with bugs. I want to make sure that doesn't happen again -- if I can help it.
I have a pretty good use case for gpt-oss. The amount of time savings has actually been wild. Definitely worth a try. Just to be clear, it gets like 2000tok/s
But it seems that there is some queuing/load balancing on their side, I mean when opus is actually outputting this 55t/s it feles fast, but apart from it's internal reasoning I think there's sometimes just waiting.
Oh wait yeah good point. At 750 tokens a second and the same amount of human patients they can set it to think for the same amount of time but four or five times the amount of thinking tokens, which may improve the quality of the eventual output.
the more advanced models also utilize a lot more tokens, and a lot of these extra tokens may go towards safeguards at a higher rate than prior models as well.
not to say a speed boost isnt there but if they didnt increase tokens / s at all youd likely see things slow down a lot with the new model compared to current
[0] https://chatjimmy.ai/