My experience has been that none of the major Deep Learning libraries (Theano, Torch7, Caffe) offer support for OpenCL, whereas they all make it trivially easy to get models running on a CUDA GPU. On top of that NVIDIA has a library of deep neural network primatives[0], and I don't believe AMD offers anything similiar.
The general consensus I've seen is to just get an NVIDIA card if you're serious about working with deep neural nets on the GPU.
One thing that did surprise me was that there was no mention of using EC2 GPU spot instances for getting your feet wet. If you don't have access to a GPU with CUDA support you can get a spot instance for about $0.07 an hour to at least test out that you have your GPU code configured correctly (and you will see some performance gains). There are even a couple of AMIs out there with Torch7 and Theano already installed.
Thanks for elaborating, these are exactly the points why AMD's GPUs are just not used in deep learning.
AWS is great if you want to use a single or two separate GPUs. However, you cannot use them for multi-GPU computation as the virtualization cripples the PCIe bandwidth; there are rather complicated hacks that improve the bandwidth, but it is still bad. Everything beyond two GPUs will not work on AWS because their interconnect is way to slow.
That was my first thought as well. I know Nvidia pretty much dominates the GPU deep learning scene, but I talked to some devs who said they got more bang for their buck using AMD GPUs. And it seems their docs have gotten significantly better over the years.
Why I did not mention AMD cards: NVIDIA's standard libraries made it very easy to establish the first deep learning libraries in CUDA, while there are no such powerful standard libraries for AMD's OpenCL. Right now, there are just no deep learning libraries for AMD cards -- so you just cannot use AMD cards for deep learning.
i feel that you should have at least mentioned it, otherwise it feels like the article implicitly asserts that only NVIDIA GPUs exist, or perhaps even that you are some way not impartial (partial..? that sounds strange).
That and Nvidia's OpenCL implementation is almost the bare minimum you can do to say you have one (unless things have changed significantly within the past year or so). Improvements to OpenCL libraries are unlikely to come from anyone already locked into using their hardware.
AMD cards were much faster than the Kepler equivalents, but now NVIDIA fixed their mistakes in that architecture and they are as fast as AMD cards. So AMDs hardware is good, it is just a software issue.
Was Kepler actually slow, or did it just seem that way because the 680 was built with their mid-range chip while the big high-end GPGPU-oriented chip was initially reserved for the server products?
Tesla GPUs are also rather slow, but the value of Tesla lies in increased double performance, larger memory and memory correction -- and you cannot beat Tesla GPUs in these departments.
Care to offer any benchmarks or an explanation of the architectural deficiencies that would make them dramatically under-perform their FLOPS and memory bandwidth ratings? When AMD cards were vastly preferred over NVidia for bitcoin mining, it had a very simple and straightforward explanation.
The general consensus I've seen is to just get an NVIDIA card if you're serious about working with deep neural nets on the GPU.
One thing that did surprise me was that there was no mention of using EC2 GPU spot instances for getting your feet wet. If you don't have access to a GPU with CUDA support you can get a spot instance for about $0.07 an hour to at least test out that you have your GPU code configured correctly (and you will see some performance gains). There are even a couple of AMIs out there with Torch7 and Theano already installed.
0.https://developer.nvidia.com/cuDNN