[Cort3z]

The magic is in the fact that they essentially have an ASIC llm device. There is no other trickery. The problem they will face is that it is actually locked in silicon, so upgrading models will be difficult, and likely require new hardware each time.

[ch4s3]

At some point I imagine you’d add a software layer on top that holds more current training and can be called as needed trading off for slower responses. There’s already work out there splitting models across networks. You could have the base on silicon, some stuff in memory on the machine, and another frontier tool in the cloud.

[jaggederest]

Taalas actually already support LoRA, basically doing exactly what you say.

The other thing that I think is really interesting about all of this, is that LLMs are already perforce behind the times with their knowledge cutoff, so adding an additional ~3 months for bake into silicon isn't such a huge deal, I think, for the ~10x more efficient and faster you get.

[ch4s3]

Yeah, really a fascinating time in computing. Once these chips start to become more common I'll be curious how people find ways to use them. One can imagine a world where really simple inference is available on dirt cheap chips found in toys and other low cost consumer electronics.

[theshrike79]

There are already studies proving that a "stupid" model with a good harness + tool calling will outperform a "smart" model.

Things like this give me hope for a system that can be fully local and private, but also with the ability to be almost infinitely extendable with tools.

[antman]

Is the size of the ASIC limiting factor or can they infinitely tensor parallelize? If thats possible then it would make it only a matter of economy of scale, there are good enough models already for people to invest in that kind of platform