|
|
|
|
|
|
by rnikander
976 days ago
|
|
|
Yes, my short post may sound dismissive, but I work in tech and I'm aware that our industry has lots of legitimate creativity and intellectual property. Of course we also have hype, marketing, BS, and herd stampedes. Some aspects of GPU/parallel design are commonly understood and other companies make such chips. So I guess I was just wondering if Nvidia had a secret sauce, are they ahead of others, and if so by what measure? I work more in software, and I'm not in a position now to evaluate their hardware on the merits. |
|
|
There were times in the past where others had a higher number of FLOPS, but it didn’t make a dent in the usage of CUDA.
My standard assumption about anything in tech (and beyond) is that you don’t win on first principles alone (“parallel design is well understood”). It’s the details that make you stand out.
Bill Daly gave a great presentation at Hot Chips a few weeks ago that explains how Nvidia’s AI performance was only in small parts due to process improvement. The rest was doing things smarter: matrix sparsity, different number representations and so forth.
https://youtu.be/rsxCZAE8QNA?si=IZneUQQ1J7Gtu81H