[qeternity]

It's not impressive nor efficient when you consider batch sizes > 1.

[p12tic]

All of this is for batch size 1.

[qeternity]

I know. That was my point.

Throughput doesn't scale on CPU as well as it does on GPU.

[p12tic]

We both agree. Batch size 1 is only relevant to people who want to run models on their own private machines. Which is the case of OP.