Ask HN: Best way to learn GPU programming?

[hazard]

I'd like to learn GPU programming but I'm having difficulty finding high-quality resources. I tried a class at coursera and was severely disappointed by both quality and content.

What are the best resources for learning things like GPU architecture, CUDA, Triton, etc?

My goal is to do be able to do something like take a description of Flash Attention and implement it from scratch, or optimize existing CUDA code.

[boberoni]

If you like textbooks, I would recommend "Programming Massively Parallel Processors" by Hwu, Kirk, and Hajj. [1] The most recent edition was published in 2022.

If you like lecture videos, I would recommend Hajj's YouTube playlist of 2021 lectures [2]. He works through a subset of the textbook.

This will give you a good foundation of GPU hardware architecture and CUDA programming. The knowledge is somewhat transferable to other areas of high-performance computing.

[1] https://www.amazon.com/Programming-Massively-Parallel-Proces...

[2] https://www.youtube.com/playlist?list=PLRRuQYjFhpmubuwx-w8X9...

[qwertox]

Thank you, that lecture is really great. Basically the second lecture shows you exactly how it's done, at least the basics with a minimal example, very well explained. Im looking forward to watch the entire playlist.

[Const-me]

For implementing stuff from scratch, if you use Windows you could try my C# based library for that: https://github.com/Const-me/Cgml/

It’s vendor agnostic, so HLSL instead of CUDA or Triton. Here’s the compute shaders implementing inference of Mistral-7B model: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral...

[Kon-Peki]

For CUDA specifically, there is a fairly large set of sample code that used to be installed when you installed CUDA. But now I think it’s on the Nvidia GitHub page; you’ve got to download it yourself.

The Nvidia dev blog has some easy to follow tutorials, but they don’t get very complicated.

Nvidia also has a learning platform which offers fairly decent courses at a cost. You get a certificate for finishing.

You’ll find some books out there with good reputations. Ultimately, this is an area that leans heavily toward paying money for good quality learning materials.

[hazard]

The best course by NVidia looks like "Fundamentals of Accelerated Computing with CUDA C/C++" which I think used to be publicly available, but is now offered "By invitation only"

[Kon-Peki]

It appears as though that course has been replaced with "Getting Started with Accelerated Computing in CUDA C/C++" which is $90 as a self-paced online class with a certificate at the end.

The course you are referring to is instructor-led and costs $10,000 to have them set up, for up to 20 people.

[the__alchemist]

Step 1: Identify a task in your project of choice that could benefit from it; ie SIMD.

Step 2: Figure out how to set up the FFI bindings if required for your project's language.

Step 3: Read this article to learn kernel syntax, block/thread/stride management etc: https://developer.nvidia.com/blog/even-easier-introduction-c...

Step 4: Ask ChatGPT to translate your code into modern C++, or perhaps even directly into Kernels

Don't bother with Vulkan compute and shaders etc. It works, but is high friction compared to CUDA.

[throwaway81523]

Disclosure: I've never done it. But I looked at some CUDA code from Leela Chess Zero and it made reasonable sense. It's just C++ with some slight changes. The GPU architecture is a little bit quirky but not that complicated either. Plus there are libraries like pytorch that handle most of the GPU stuff for you.

I would say ML concepts and algorithms are way more complicated than GPU programming per se. The fast.ai lectures were pretty understandable when I watched some of them a few years ago, but attention wasn't yet invented, and it was pretty obvious that it would take a fair amount of trial and error to become skilful at writing simple recognizers.

[Baldbvrhunter]

It would also mean learning Julia, but you can write GPU kernels in Julia and then compile for NVidia CUDA, AMD ROCm or IBM oneAPI.

https://juliagpu.org/

I've written CUDA kernels and I knew nothing about it going in.

[madsciencetist]

While I am a fan of Julia and its GPU module, using such an easy environment will really limit what you are able to learn. NVIDIA provides some great optimization tools (NSight Systems and NSight Compute) which help you optimize your kernel execution (fuse kernels, hide latency, use execution graphs) and CUDA code (take advantage of memory layout, use warp intrinsics, maximize throughput). These tools map to C++/CUDA source code and let you rapidly address bottlenecks - most of which may be on the host side.