[37ef_ced3]

AVX-512 neural net inference on inexpensive, CPU-only cloud compute instances. GPU cloud compute is almost unbelievably expensive. Even Linode charges $1000 per month, or $1.50 per hour (look at the GPU plans: https://www.linode.com/pricing/#row--compute)

An AVX-512 Skylake-X cloud compute instance costs $10 per CPU-core per month at Vultr (https://www.vultr.com/products/cloud-compute/), and you can do about 18 DenseNet121 inferences per CPU-core per second (in series, not batched) using tools like https://NN-512.com

As AVX-512 becomes better supported by Intel and AMD chips, it becomes more attractive as an alternative to expensive GPU instances for workloads with small amounts of inference mixed with other computation

[rewq4321]

How could it be competitive with GPUs in terms of price-per-unit-performance? Seems like GPUs are expensive only because you can't rent a small portion of a GPU? But shouldn't that be possible with GPU virtualisation?

Or is it the case that if you virtualised a GPU up into tiny pieces, the memory-to-flops ratio would be way off what's needed for inference? Or the virtualisation overhead would be too big?

Those are all genuine questions, just to be clear - this is not my area of expertise.

[37ef_ced3]

Well, a GPU is a cluster of SIMD units with fast memory

A GPU thread variable is just like a SIMD lane, and a GPU warp variable is just like a SIMD vector register

Nvidia's SIMD instructions are called PTX and they are similar to AVX-512

An AVX-512 core is like a general purpose CPU with a 512-bit GPU core built in

So paying for a single AVX-512 core is like paying for part of a GPU, plus the general purpose compute you need to keep the GPU supplied with work

If you could divide the GPU up, you would lose most of the parallelism, keep all of the communication latency, and still need the drivers etc.

Would a hypothetical virtualized GPU be competitive with an AVX-512 core in terms of price/performance? I don't know, I haven't done the comparison