[exabrial]

Why don't we have LLM-Accelerator cards (yet)? Just seems like graphics cards are a good choice, but not a purpose-built choice.

[cjbprime]

Inference (not training) is bottlenecked by memory access speed, not compute. Having special hardware wouldn't make it faster unless you somehow found a faster memory controller than the GPU has.