[quotemstr]

What precisely is an ASIC supposed to do that a more programmable accelerator can't? Memory latency is memory latency: doesn't matter whether it's embedded in a cache-line wait state or some flipflop state machine. ROM isn't going to be faster than RAM either.

Likewise, for compute, is the ASIC somehow going to beat a systolic array? You can't have one circuit per weight: the die area and electrical fan-out would be insane. I'm not seeing how an ASIC specialized for a specific model would actually help much.

I mean, sure, we can build more specific accelerators, e.g. for softmax, but these work fine in the context of a programmable pipeline.

Yes, there are more exotic things out there, like optical matrix multiplication systems. Those are different. But above, aren't you talking about just doing conventional digital linear algebra, but with a model-specific set of circuits?

[marcyb5st]

No, if you burn your model into the silicon you don't need memory as the output of a layer flows through circuitry directly into the next one. No I/O to memory of any kind. You still need a bit of memory for in flight answers, but that's it.

[quotemstr]

You can pipeline in software too, and even with a dedicated circuit, you need to put weights somewhere, because you're not going to have a special partially-applied FP8 FMA unit for each of your trillion model weights.

I can accept the idea of specializing a circuit for a specific model shape, but I'm not seeing a need to specialize a circuit for the weights inside the shape.

[marcyb5st]

Apparently that is what Taalas did though. Not an hardware person, so take this with a pinch of salt