[gymbeaux]

This is huge. Anyone who was considering AMD + ROCm as an alternative to NVIDIA in the AI space isn’t anymore.

I’m one of those people who can’t (won’t) learn C++ to the extent required to effectively write code for GPU execution…. But to have a direct pipeline to the GPU via Python. Wow.

The efficiency implications are huge, not just for Python libraries like PyTorch, but also anything we write that runs on an NVIDIA GPU.

I love seeing anything that improves efficiency because we are constantly hearing about how many nuclear power plants OpenAI and Google are going to need to power all their GPUs.

[ferguess_k]

Just curious why can't AMD do the same thing?

[bigyabai]

It can be argued that they already did. AMD and Apple worked with Khronos to build OpenCL as a general competitor. The industry didn't come together to support it though, and eventually major stakeholders abandoned it altogether. Those ~10 wasted years were spent on Nvidia's side refining their software offerings and redesigning their GPU architecture to prioritize AI performance over raster optimization. Meanwhile Apple and AMD were pulling the rope in the opposite direction, trying to optimize raster performance at all costs.

This means that Nvidia is selling a relatively unique architecture with a fully-developed SDK, industry buy-in and relevant market demand. Getting AMD up to the same spot would force them to reevaluate their priorities and demand a clean-slate architecture to-boot.

[pjmlp]

Maybe because Apple got pissed on how Khronos took over OpenCL, AMD and Intel never offered tooling on par with CUDA in terms of IDE integration, graphical debuggers and library ecosystem.

Khronos also never saw the need to support a polyglot ecosystem with C++, Fortran and anything else that the industry could feel like using on a GPU.

When Khronos finally remember to at least add C++ support and SPIR, again Intel and AMD failed to deliver, and OpenCL 3.0 is basically OpenCL 1.0 rebranded.

Followed by SYCL efforts, which only Intel seems to care, with their own extensions on top via DPC++, nowadays openAPI. And only after acquiring Codeplay, which was actually the first company to deliver on SYCL tooling.

However contrary to AMD, at least Intel does get that unless everyone gets to play with their software stack, no one will bother to actually learn it.

[bigyabai]

Well, Apple has done nothing to replace the common standard they abandoned. They failed to develop their proprietary alternatives into a competitive position and now can't even use their own TSMC dies (imported at great expense) for training: https://www.eteknix.com/apple-set-to-invest-1-billion-in-nvi...

However you want to paint the picture today, you can't say the industry didn't try to resist CUDA. The stakeholders shot each other in a 4-way Mexican standoff, and Nvidia whistled showtunes all the way to the bank. If OpenCL was treated with the same importance Vulkan was, we might see a very different market today.

[pjmlp]

Yes they did, it is called Metal Compute, and everyone using Apple devices has to use it.

Vulkan you say?

It is only relevant on GNU/Linux and Android, because Google is pushing it, and still most folks still keep using OpenGL ES, no one else cares about it, and already turned into the same spaghetti mess as OpenGL, to the point that there was a roadmap talk at Vulkanised 2025 on how to sort things out.

NVidia and AMD keep designing their cards with Microsoft for DirectX first, and Vulkan, eventually.

[jms55]

> NVidia and AMD keep designing their cards with Microsoft for DirectX first, and Vulkan, eventually.

Not really. For instance NVIDIA released day 1 Vulkan extensions for their new raytracing and neural net tech (VK_NV_cluster_acceleration_structure, VK_NV_partitioned_tlas, VK_NV_cooperative_vector), as well as equivalent NVAPI extensions for DirectX12. Equal support, although DirectX12 is technically worse as you need to use NVAPI and rely on a prerelease version of DXC, as unlike Vulkan and SPIR-V, DirectX12 has no mechanism for vendor-specific extensions (for good or bad).

Meanwhile the APIs, both at a surface level and how the driver implements them under the hood, are basically identical. So identical in fact, that NVIDIA has the nvrhi project which provides a thin wrapper over Vulkan/DirectX12 so that you can run on multiple platforms via one API.

[bigyabai]

> it is called Metal Compute, and everyone using Apple devices has to use it.

Sounds like a submarket absolutely teeming with competition. Like, you have Metal Compute, and Apple Accelerate Framework and MLX all sitting there in the same spot! Apple is really outdoing themselves, albeit in a fairly literal sense.

> It is only relevant on GNU/Linux and Android

Hmm... someone ought to remind me of the first stage of grief, I've forgotten it suddenly.

[saagarjha]

They did; they are active contributors to OpenAI’s Triton compiler which has a very similar execution model.

[dismalaf]

> But to have a direct pipeline to the GPU via Python

Have you ever used a GPU API (CUDA, OpenCL, OpenGL, Vulkan, etc...) with a scripting language?

It's cool that Nvidia made a bit of an ecosystem around it but it won't replace C++ or Fortran and you can't simply drop in "normal" Python code and have it run on the GPU. CUDA is still fundamentally it's own thing.

There's also been CUDA bindings to scripting languages for at least 15 years... Most people will probably still use Torch or higher level things built on top of it.

Also, here's Nvidia's own advertisement and some instructions for Python on their GPUs:

- https://developer.nvidia.com/cuda-python

- https://developer.nvidia.com/how-to-cuda-python

Reality is kind of boring, and the article posted here is just clickbait.

[dragonwriter]

> It's cool that Nvidia made a bit of an ecosystem around it but it won't replace C++ or Fortran and you can't simply drop in "normal" Python code and have it run on the GPU.

While its not exactly normal Python code, there are Python libraries that allow writing GPU kernels in internal DSLs that are normal-ish Python (e.g., Numba for CUDA specifically via the @cuda.jit decorator; or Taichi which has multiple backends supporting the same application code—Vulkan, Metal, CUDA, OpenGL, OpenGL ES, and CPU.)

Apparently, nVidia is now doing this first party in CUDA Python, including adding a new paradigm for CUDA code (CuTile) that is going to be in Python before C++; possibly trying to get ahead of things like Taichi (which, because it is cross-platform, commoditizes the underlying GPU).

> Also, here's Nvidia's own advertisement for Python on their GPUs

That (and the documentation linked there) does not address the new upcoming native functionality announced at GTC; existing CUDA Python has kernels written in C++ in inline strings.

[freeone3000]

OpenCL and OpenGL are basically already scripting languages that you happen to type into a C compiler. The CUDA advantage was actually having meaningful types and compilation errors, without the intense boilerplate of Vulkan. But this is 100% a python-for-CUDA-C replacement on the GPU, for people who prefer a slightly different bracketing syntax.

[dismalaf]

> But this is 100% a python-for-CUDA-C replacement on the GPU

Ish. It's a Python maths library made by Nvidia, an eDSL and a collection of curated libraries. It's not significantly different than stuff like Numpy, Triton, etc..., apart from being made by Nvidia and bundled with their tools.

[gymbeaux]

I’m mainly interested in the performance implications. The less shit between me and the hardware, theoretically the better the performance. In a world where these companies want to build nuclear power plants just to power NVIDIA GPU data centers, I feel like we need to be optimizing the code where possible.

[pjmlp]

Yes, shading languages which are more productive without the gotchas from those languages, as they were designed from the ground up for compute devices.

The polyglot nature of CUDA is one of the plus points versus the original "we do only C99 dialect around here" from OpenCL, until it was too late.

[ErrorNoBrain]

They are, if they cant find an nvidia card

[pjmlp]

NVidia cards are everywhere, the biggest difference to AMD is that even my lousy laptop GeForce cards can be used for CUDA.

No need for a RTX for learning and getting into CUDA programming.

[gymbeaux]

True, although I believe Maxwell is the oldest supported architecture for the current CUDA 12.x. Maxwell (eg GTX 980) came out around 2013, if memory serves. 10+ years of support is not bad at all considering ROCm supports only like 3 consumer AMD GPUs.

So your lousy laptop GTX 750Ti ehhh probably can’t practically be used for CUDA. But your lousy 1050Ti Max-Q? Sure.