[mac01021]

Question:

What are the characteristics of NVIDIA GPUs that make them superior for deep learning applications?

Phrased another way, if you're designing a card specifically to be good for training deep neural nets, how does it come out differently from cards designed to be good at other typical applications of GPGPU?

[lightcatcher]

Nvidia's main advantage over AMD for deep learning is software, not the hardware itself. Nvidia's software advantage has lead to network effects where almost all GPU supporting deep learning libraries build on top of Nvidia. There's no Tensorflow/Theano/Torch/Keras/etc for AMD/OpenCL for the most part. Nvidia also releases high performance kernels for common neural net operations (such as matrix multiply and convolution).

On the hardware end, Nvidia has slightly superior floating point performance (which is the only thing that matters for neural nets). Pascal also contains 16 bit floating point instructions, which will also be a big boost to neural net training performance.

[mac01021]

That makes a lot of sense about the software.

I would be interested to hear about the difference in floating point performance. I would have guessed that, at this point, pretty much every chip designer in the world knows equally well how to make a floating-point multiplier. So it must be that nvidia vs amd have made different trade-offs in when it comes to caching memory near the FP registers or some such thing?

[lightcatcher]

I'm unsure about the floating point performance differences, but 2 other reasons for potential differences are (1) number of floating point units and (2) different clock speeds.

[hooloovoo_zoo]

It doesn't hurt that the Titan X and it's professional counterpart have 12 GB of VRAM.