[rerx]

GPUs will be better optimized for large matrix multiplications than CPUs, by design.

And you need to the inference again and again, not just a single time (like your training).

[dist-epoch]

Desktop CPUs have integrated GPUs, so it's more complicated. If I infer on the GPU inside my CPU, how do you count that?

[rerx]

True. Memory bandwidth may be the most limiting factor.