[solarkraft]

I used to be so bullish on Cerebras, being pretty certain that specialized chips would eventually dethrone NVidia for inference hardware.

Multiple years have passed since then. GPT spark got me excited again, but somehow that seems to have faded right back into obscurity.

Can somebody explain why there’s so little apparent progress here despite the theoretically massive advantage? Can I still expect this to happen eventually?

[phonon]

Because Cerebras handles large models poorly due to latency/bandwidth issues to main memory. See https://openai.com/index/introducing-gpt-5-3-codex-spark/ where its performance is significantly below that of the regular Codex 5.3, and can only handle a 128k text context window. For some use cases its great, but most would rather use a better, slower model.

In the future, they plan hybrid implementations, to be able to serve large models better, e.g.

"AWS. We signed a binding term sheet with Amazon Web Services for AWS to become the first hyperscaler to deploy Cerebras systems in its data centers. Deployment in AWS data centers will require us to meet strict standards for performance, scale, and reliability.Pursuant to the term sheet, we will create a co-designed, disaggregated inference-serving solution that will integrate AWS Trainium3 chips with Cerebras CS-3 systems, connected via high-bandwidth networking, to partition inference workloads across Trainium3 and CS-3. Each system will perform the type of computation at which it most excels. The approach is expected to deliver 5 times more token throughput in the same hardware footprint, at up to 15 times faster speeds compared to leading GPU-based solutions as benchmarked on leading open-source models."

[ismailmaj]

Unclear if it's the only cause but wafer scale is great for very low latency, but loses to throughput per dollar compared to classic Nvidia like GPUs. I don't think they can reduce the gap, SRAM is just more expensive than HBM and their architecture needs a lot of it.

So, the price makes it necessarily niche to some specific use-cases like HFT or intelligent duplex voice assistants, I'm still semi-bullish personally.

[IndignantTyrant]

Taalas seems to be pretty good. This is their demo: https://chatjimmy.ai/

[pants2]

This might be even more limited though. They can't physically fit a large model on a single chip.

[imdoxxingme]

Not yet. But it is definitely limited, since they can only serve a single model essentially.

[gpapilion]

The initial cost of serving is very high, and while super performant not great for scaling up.

In practice they are also not very flexible when compared to gpus.