ML researchers in statistics departments write stuff in R, which makes everyone scream. ML researchers absolutely do.
My point in the article was basically the class was "indoctrinating" (too strong, but you get the point) the future ML researchers in the superiority of using CUDA and spending NVIDIA company resources to continuously do so in these classes, year after year.
This hits the nail on the head. Nvidia got all the programmers excited about using their GPUs first and now they have all the software targeting their hardware.
If you could compile CUDA for Intel and AMD it's not going to perform well. When you program a GPU you aren't just writing task specific code, you are also writing hardware specific code. So having developer mindshare matters much more than having a nice programming language.
In ML many people write pytorch and not CUDA. But even in ML the choice of precision is driven by the data types Nvidia can deal with efficiently - this is a moat which is nothing to do with CUDA.
Yes. Writing CUDA that calls CuBLAS or CUB is still writing CUDA. Lots of kernels and functions (functors, etc) are "business requirements" moreso than math libraries. It's no different than the CPU code world, there are far more CRUD apps than BLAS libraries written, and writing a CRUD app that calls a BLAS library doesn't mean you're not "writing CPU code". Someone has to write those systems of linear equations for BLAS to solve.
The world is deeper than just assembly and BLAS tuning, and you can get extremely far in CUDA just by gluing together the primitives they give. Python is popular in the AI/ML space, but far from the only way to do that.
Brought down some simulations from about 30min to under 1s.