[pclmulqdq]

You would be surprised. The GPU often loses even for small neural nets given the large latency. Anything that needs high throughput or is sized like an HPC problem should use a GPU, but a lot of code benefits from SIMD on small problems.

[gmueckl]

If you run many small tasks on the GPU, you can increase throughput by overlapping transfers and computation. There may also be other ways to batch problems together, but that depends on the algorithms.

The one truly unfixable issue is round-trip latency.

[gopalv]

> The GPU often loses even for small neural nets given the large latency

Apple's neural engine shows that you can live in between those two worlds.

As you said, the trouble is the latency, the programming model is still great.