[AlexCoventry]

Hmm, I still have some nonrefundable API credits with OpenAI. Maybe I should try to use them for my kernel.

FWIW, this talk[1] from NVIDIA/Meta from March claims that coding agents can often write correct implementations of of CUDA kernels, but that they're usually dog slow, like 100x slower than a kernel optimized by a skilled human.

[1] https://www.nvidia.com/en-us/on-demand/session/gtc26-s81653/

[osti]

For me it was able to try out different architectures for perf improvement, then once it's settled on some good architectures, it can do lower level optimizations on them by profiling the code etc.

[AlexCoventry]

That's great. It seems like a large body of experts are having real problem with this, so maybe you should publish something about your methods, or start a business...

[osti]

I can't vouch for whether or not it can beat human experts though because I'm no CUDA expert myself. The original CUDA code were human written and I first let codex adapt it to my specific use case. Then I basically let codex generate ideas and try the ideas out itself (I think it's a bit like Karpathy's autoresearch, except I was still doing manual prompting). And that was enough to get me 20x improvement.

I suspect when people said AI wrote non performant CUDA kernels it was beginning-mid last year and it's definitely vastly improved since back then. And the agent's ability to iteratively improve really impressed me.