|
I agree w/ you about moving up a gradient quickly w/ the "GPU manufacturing <-> deep learning research" feedback loop. I think it could last a while though. One really important area of research is figuring out how to take better advantage of greater capacity. Also, how to do more with fewer training samples (0 shot, 1 shot, etc learning). Then there's reducing the precision of the units you're using to increase capacity through software. Applying these algos to video, audio, media generation, and others will eat up all the resources you can throw at it; the algos today could take advantage of larger capacity when applied to time-series. There's so much going on that I don't see it slowing down for at least 3-5 years. Also, I'd like to point out that we've seen some big breakthroughs in the past 10 years. But for the past 10 years, the whole field of deep learning has been looked at with skepticism and has been very niche. Over the past couple of years, a lot of money and resources have been put in place to pursue this area of research. More money doesn't necessarily mean more results, but there are many many more people working on these problems than ever before. A lot of them are legitimately brilliant researchers in the prime of their careers. I think there's still more to see. I am concerned about an Nvidia monopoly around deep learning hardware. They give away tons of free cards to deep learning research groups, but at some point they'll want people to start buying. I assume they expect that will be the enterprise set, but if suddenly they manage to move all their capable deep learning cards to Teslas only (which have a huge markup), it will put the hobbyist deep learning developer at a big disadvantage. The only check and balance on that is the fact that Nvidia makes cards for gamers too, and gaming card competition is still somewhat robust, so any technology that gives enterprise a big boost will probably make its way to their flagship gaming cards quickly. Nvidia's only real competition, AMD, is so far behind that they might as well not be in the business. As someone who usually roots for the underdog, it pains me to see AMD fumble so badly in this whole area. Quantum computing could be a new hill, but I think that's a ways out and I don't know enough about the topic to speak with any real confidence. |
1. GPU as a service: While all major providers (AWS, Azure) offer Teslas on their servers, there is no AMD on the cloud (that I know of)
2. Key libraries: Nvidia comes with matrix libraries and cuDNN out of the box, while for AMD, there are only open source offerings that are a bit difficult to manage.
But, If you write your own software or rely on open source, AMD is quite performant and affordable. The problem is that it is really obscure. So, yes, they are really bad at marketing and if you are looking at them as a user instead of as a developer, they are invisible.