[hazz99]

Do you have any career advice for someone deeply interested in breaking into high performance GPU programming? I find resources like these, and projects like OpenAIs Triton compiler or MIMD-on-GPU so incredibly interesting.

But I have no idea who employs those skills! Beyond scientific HPC groups or ML research teams anyway - I doubt they’d accept someone without a PhD.

My current gameplan is getting through “Professional CUDA C programming” and various computer architecture textbooks, and seeing if that’s enough.

[pjmlp]

Given that CUDA main focus is C++ since CUDA 3.0, ignoring the other PTX sources for now, not sure if that 2014 book is the right approach to learn CUDA.

[anonymousDan]

Can you elaborate a bit on how C++ affects the programming model? Isn't CUDA just a variant of C? I presume it is not the goal to run standard C++? Also as I understand it PTX is an IR so not sure why C/C++ can be compared?

[pjmlp]

Not at all, unless we are speaking of CUDA until version 3.0.

CUDA is a polyglot programming model for NVidia GPU, with first party support for C, C++, Fortran, and anything else that can target PTX bytecode.

PTX allows for many other languages with toolchains to also target CUDA in some form, with .NET, Java, Haskell, Julia, Python having some kind of NVidia sponsored implementations.

https://developer.nvidia.com/language-solutions

While originally CUDA had its own hardware memory model, NVidia decided to make it follow C++11 memory semantics and went through a decade of hardware redesign to make it possible.

- CppCon 2017: Olivier Giroux "Designing (New) C++ Hardware”

https://www.youtube.com/watch?v=86seb-iZCnI

- The CUDA C++ Standard Library

https://www.youtube.com/watch?v=g78qaeBrPl8

It is also driving many of the use cases in parallel programming for C++

- Future of Standard and CUDA C++

https://www.youtube.com/watch?v=wtsnoUDFmWw

You will only find brief mentions of C here,

https://developer.nvidia.com/hpc-compilers

This is why OpenCL kind of lost the race, with it focused too much in its C dialect, only going polyglot when it was too late for the research community to care.