Taalas claims significant cost, speed, and power differential with running on Nvidia (and similar) hardware. Their whole thesis is that with making the hardware cost less, run faster, and run with less power mixed with their claimed fast production cycle for new weights, companies would plan on purchasing and then relegating the other chips to different workloads at the next update cycle. And the chips have facilities for fine-tunes that can by dynamically loaded. The info I've seen has been very light on details, but if they can even get halfway where they are planning, it'll be a massive shift.
Interesting. Is the speedup from specializing for the shape of Llama 3.1 or are they (contra my mental model) actually winning on burning in the weights?
Taalas only uses SRAM for the KV cache and the activations, the weights are in mask rom in the metal layers.
If they designed this right, it means that once they have a model, so long as they keep the hyperparameters fixed they can change the weights much faster than it takes to spin up a completely new chip, essentially at a cost of doing a minor revision.