|
> As someone who works at a deep learning chip startup, this is great news! Looks like there's a market for our chips ;) While there may be a market for your chips, I'm curious why you think the K computer is in that market? National supercomputers, like Japan’s RIKEN "K" supercomputer, are used for many different applications (for example, physics and engineering simulations) - not just "AI." The multi-purpose use of such machines is what justifies their multi-billion dollar budgets in the first place. I can't imagine a government spending billions of dollars on a machine that only has one function (e.g. neural net training). The history of HPC hardware is littered with special-purpose HPC microarchitectures that were eventually abandoned in favor of general-purpose processors. The one lasting exception to this has been GPUs, which have proven to be a boon to HPC applications and sparked the Deep Learning renaissance in machine learning. The difference with GPUs is that they were not strictly aimed at HPC applications. Obviously, they are used for graphics rendering in gaming, professional graphics and CAD. There are hundreds of millions of GPUs deployed for gaming and other graphics applications. The application of GPUs to HPC came later, and the specific application to deep neural networks came later still. GPUs are successful because they are a form of commodity hardware and have a wide range of applications. In a sense, hard-core gamers have become the R&D funding source for state-of-the-art HPC processors. This healthy and diversified ecosystem is what allows for the long-term sustainability of the microarchitecture. You can always build a more efficient machine by specializing it to a narrow application. In the extreme case, you can just build a custom ASIC that has some fixed function. That would be the ultimate solution in efficiency, but things become less sustainable when you need to continuously compete with alternative solutions - the cost of competing in this space is astronomical, and there needs to be sustainable source of funding for that activity. This is why the HPC industry is completely dominated by Intel/AMD/NVIDIA processors, instead of custom ASICs that (for example) could perform some fixed matrix operations. Having said that, there is a vague opportunity on the horizon if and when Moore's Law scaling completely fizzles out. Conceivably, after processor node scaling completely ends and the established microarchitectures have been completely optimized to death, the industry will reach a state where competition on performance/efficiency stalls because there is no major next-gen CPU or GPU because nothing more can be done to improve the product while maintaining its general-purpose applicability. At that stage, a significant opportunity could open up for special-purpose processors and it could be sustainable since the field would be far less competitive. |
For the record, many of the improvements we were able to squeeze out for deep learning has also lead us to create two new designs, one for a GPU and one for a CPU. Although we're focusing on the deep learning processor for now, the ultimate goal is to develop all three and put them on a SoC. This is too ambitious in our current stage, and so we're focusing on the deep learning processor.