[api]

I had a thought a while back: sell large local models burned onto fused compute / ROM chips. Like cartridges for old game consoles. Slot (or probably plug into USB-C) and go.

It’s an ASIC with the model wired into it so it’s very low power and fast.

I’d buy these. Say $100 for a frontier class model. Maybe more.

[tybit]

Taalas is developing this, but not for Frontier class models. I hope that if we can least get the easy 80% of work done on that sort of hardware, we can greatly reduce the demand for GPUs, HBM and energy to some extent.

[wild_egg]

There is an amount of brute forcing that becomes possible at those speeds that I think could even take us beyond 80%. If we could have Qwen3.6-27B running at 15k t/s, run 100 attempts concurrently, select top-K solutions and synthesize a final result from them.

There was a paper a while back that showed top-K selection like that with tiny models was able to reliably solve some 1M-step Tower of Hanoi when no frontier model could. Very big level up in capability just from horizontally scaling compute.

[yomismoaqui]

100 dumb folks don't make an Einstein

[wild_egg]

You pull out Einstein when you need a breakthrough.

[Schlagbohrer]

To extend this metaphor or the "million monkeys typing at a million keyboards", it occurs to me that if you can just slightly increase the intelligence of each sub-agent by a tiny amount, the overall end product can shoot up in quality tremendously.

If I can make my small fast AI model just a tiny, tiny bit more capable, and still run 100 of them or 1000 and run an evaluation model on top of that, the overall system capability will scale quickly with tiny increases in base model intelligence.

[mdp2021]

But some (Meta, Anthropic) suggested that optimizing and extending the "<think>" process can produce extra value. (I do not know if that requires an improved underlying architecture - frontier models architectures are sometimes not public.)

[jjcm]

They've unfortunately been radio silent since their release, and aren't active on socials at all. I've tried to contact them for updates / api usage, but haven't heard anything back.

[i5heu]

Looking at their career page it looks like they do not care that much about PR at the moment... considering that they have a live chat with 14000t/s via Llama 3.1 8B[0] i don't really think they need to do PR either.

So i guess maybe they currently try to solve a very hard problem with a small focused group before scaling or they are dysfunctional.

Also Llama 3.1 8B is a dense model AFAIK and they are fast by nature. As there are not a lot of dense models these days i could imagine that they try to optimise for MOE models.

[0] https://chatjimmy.ai/

[glaslong]

I love this for the popular sci-fi trope too, where you see some ship engineer swap one glowing crystal "compute core" for another.

We could have the photonic AI model ASICs for real!

[api]

In Gibson’s Neuromancer AIs burned onto chips were literally discussed and called “ROM constructs.” In that case some of them were images of human minds but with no persistent state or memory.

So basically a static model version of consciousness uploading.

[jazzyjackson]

Or Galatea’s personality chip in Bicentennial Man :)

[theturtletalks]

Taalas[0] seems to be what you're talking about.

0. https://taalas.com/

[AussieWog93]

Is suspect you're looking at more like $100k than $100 for an ASIC with enough memory and compute to handle Fable.

But the flip side is possibly 1000+ tok/s on a SOTA model, which would be game changing.

Could make sense for datacentres or enterprise, but I don't think we'll be getting SOTA Game Boy carts this decade.

[TiredOfLife]

Kinda already exists.

demo https://chatjimmy.ai/

https://news.ycombinator.com/item?id=47103661

[ktzar]

Hallucinates on the first question I ask, as 90% of these models that try to take shortcuts.

[phildenhoff]

You’re expecting the wrong thing. The demo demonstrates the insane inference rate of dedicated hardware. Iirc it’s llama 3 or something. Not a very good model by today’s standards. But it runs at 16k tokens per second, an order of magnitude above the competition.

Imagine what’s possible if you had GLM-5.2 turned into a hardware chip like this.

[versteegen]

It's a custom 3 bit quant of Llama 3.1 8B and other shortcuts. The quant is not good. Their newer arch switches to standard 4 bit quants, should be far better!

[LarsDu88]

I believes the weights are burned as ROM microcode, but for an effective inference speedup, you do want to burn the architecture (matmuls, activation functions, MoE gates, etc) as well which will differ from model to model.

It's not as simple as a weight swap between identical architectures.

The speed gains are also from not having to route the weights through wiring like with ROM cartridges.

[smokel]

> I’d buy these. Say $100 for a frontier class model. Maybe more.

Sure you would. Running frontier class models on current hardware costs in the order of tens of thousands of dollars. It is more likely that these custom ASICs will be priced competitively with that, and not with Super Mario Bros.

Oh, and energy consumption will be in the same order.

[glimshe]

Interestingly, you could easily run them from the said old consoles! You'd just need a bit of console code to interface (text input/output) with your fully independent LLM subsystem. Imagine Claude for the NES without Internet?

[Gecko4072]

This would be very compelling. Can anyone share more details on how it would work? Only issue is that you are stuck at a certain point in time but that’s not a huge deal. Even just a good 27b model would be useful.

[FridgeSeal]

Talaas have done this with a llama 3 model. Runs at like, 16k/tokens a second oror something obscene. Very little power draw too.

Doesn’t need hbm or lots of memory, because the hardware can just forward the data straight to the next layer and you don’t need to round trip through memory.

They claim to be working on an approach to make the underlying hardware a bit more reusable between models.

[foobar10000]

Yeah, if you have a fixed llm topology, you can just effectively burns 2 top layers of the chip as Rom (model weights) - which has a per area density even better than dram - so it’s just attention and kv streaming that is hbm to sram transfer.

Most big model weights will not fit a single reticle sized chip - so you’d have prob 30 different chips to split the model .

And you’d need super fast chip to chip comms for the all-reduce and similar.

So scaling to 1T models is hard - and a long lead time - but can be very power efficient.

[i5heu]

Looks like 27b models are in reach.

> “In the current generation, our density is 8 billion parameters on the hard wired part of the chip., plus the SRAM to allow us to do KV caches, adaptations like fine tuning, and etc. In our next generation, we would have the ability to go up to 20 billion parameters in a chip. Even with trillions of parameters, we’re talking about few tens of chips, which is a very, very small compared to anything else out there on the market today.”

https://www.nextplatform.com/compute/2026/02/19/taalas-etche...

Edit: i do not know how reliable this page is... it has a lot of typos.. for me it does not look like it was written by a LLM

[all2]

There a lot of ways this could work.

1) the hardest, custom silicon + MCU to manage the USB interface

2) not as hard, shared memory, NPU + MCU to manage inference and USB interface

Theoretically you could do 2 with the right MCU, NPU, and memory combo. You'd stream/DMA the weights from memory into the NPU and then read the results with the MCU. From a user's perspective, it might take the form of an openAI API compatible endpoint that enumerates when they plug the USB device in. There would likely be some host-side software to ease the pain of trying to use a USB device as an HTTP API.

[Schlagbohrer]

This is a fun idea but they'll be outdated in 6 months and it would take 12 months to develop, build and ship them assuming they can design and manufacture these at superhuman speeds.

[taegee]

https://chatjimmy.ai

[tomaskafka]

Sounds like something DARPA would be working on right now.

[Cantinflas]

You need terabytes of memory to run a frontier class model

[calgoo]

I wonder, if you can run at 8k or 15k t/s, you could in theory run 10 or 20 agents (or more) at the same time and generate hundreds of versions, then just analyze them. Think thinking mode x1000 at least... Would be interesting to see how good it would be

[est]

If you look closely, 80% of those terabytes of memory is stationary model weights.

[matheusmoreira]

I'd absolutely buy these chips as well.

[yieldcrv]

Thats happening, in progress

We’ll see what the market chooses

[fendy3002]

Your statement reminds me of Avenger's scene where Tony choosing Friday among other AI's catridges to use.

That sounds good and practical to happen!

[k33P1Tr3aL]

isn't this essentially an FPGA?

[whywhywhywhy]

FPGA is reprogrammable, ASIC only does one thing forever.

[popalchemist]

How very Cyberdyne.

[jdw64]

Wow, I'd really love it if that were the case. I'm already pretty satisfied with just GPT 5.6 as it is.