Taalas is developing this, but not for Frontier class models. I hope that if we can least get the easy 80% of work done on that sort of hardware, we can greatly reduce the demand for GPUs, HBM and energy to some extent.
There is an amount of brute forcing that becomes possible at those speeds that I think could even take us beyond 80%. If we could have Qwen3.6-27B running at 15k t/s, run 100 attempts concurrently, select top-K solutions and synthesize a final result from them.
There was a paper a while back that showed top-K selection like that with tiny models was able to reliably solve some 1M-step Tower of Hanoi when no frontier model could. Very big level up in capability just from horizontally scaling compute.
To extend this metaphor or the "million monkeys typing at a million keyboards", it occurs to me that if you can just slightly increase the intelligence of each sub-agent by a tiny amount, the overall end product can shoot up in quality tremendously.
If I can make my small fast AI model just a tiny, tiny bit more capable, and still run 100 of them or 1000 and run an evaluation model on top of that, the overall system capability will scale quickly with tiny increases in base model intelligence.
But some (Meta, Anthropic) suggested that optimizing and extending the "<think>" process can produce extra value. (I do not know if that requires an improved underlying architecture - frontier models architectures are sometimes not public.)
They've unfortunately been radio silent since their release, and aren't active on socials at all. I've tried to contact them for updates / api usage, but haven't heard anything back.
Looking at their career page it looks like they do not care that much about PR at the moment... considering that they have a live chat with 14000t/s via Llama 3.1 8B[0] i don't really think they need to do PR either.
So i guess maybe they currently try to solve a very hard problem with a small focused group before scaling or they are dysfunctional.
Also Llama 3.1 8B is a dense model AFAIK and they are fast by nature.
As there are not a lot of dense models these days i could imagine that they try to optimise for MOE models.
There was a paper a while back that showed top-K selection like that with tiny models was able to reliably solve some 1M-step Tower of Hanoi when no frontier model could. Very big level up in capability just from horizontally scaling compute.