[singhrac]

I've heard credible claims that GPUs these days (esp. TPUs) have lower latency for big models than CPUs. I haven't really investigated, but I could see it happening if you give the TPU a huge L1 cache or something.

[someguydave]

Perhaps for large calculations? Otherwise the PCI transfer delay would be a big latency hit?

[adamnemecek]

Yeah until TPUs can directly communicate with the sound card, it sounds slow.