[metadat]

But how will AMD or anyone else push in? CUDA is actually a whole virtualization layer on top of the hardware and isn't easily replicable, Nvidia has been at it for 17 years.

You are right, eventually something's gotta give. The path for this next leg isn't yet apparent to me.

P.s. how much is an exaflop or petaflop, and how significant is it? The numbers thrown around in this article don't mean anything to me. Is this new cluster way more powerful than the last top?

[vlovich123]

The API part isn't thaaat hard. Indeed HIP already works pretty well at getting existing CUDA code to work unmodified on AMD HW. The bigger challenge is that the AMD and Nvidia architectures are so different that the optimization choices for what the kernels would look like are more different between Nvidia and AMD than they would be between Intel and AMD in CPU land even including SIMD.

[pjmlp]

Only if the only thing one cares about is CUDA C++, and not CUDA C, CUDA C++, CUDA Fortran, CUDA Anything PTX, plus libraries, IDE integration, GPU graphical debugging.

[vlovich123]

CUDA C works fine with HIP not sure what you're referring to. As for the other pieces, GPU graphical debugging isn't relevant for CUDA and I don't know what IDE integration is special / relevant for CUDA but AMD does have a ROCm debugger which I would imagine would be sufficient for simultaneous debugging of CPU & GPU. You won't get developer tools like nsight systems but I'm pretty sure AMD has equivalent tooling.

As for Fortran, that doesn't come up much in modern AI stuff. I haven't observed PTX / GCN assembly within AI codebases but maybe you have extra insight there.

[vitus]

> P.s. how much is an exaflop or petaflop, and how significant is it? The numbers thrown around in this article don't mean anything to me. Is this new cluster way more powerful than the last top?

Nominally, a measurement in "flops" is how many (typically 32-bit) FLoating-point Operations Per Second the hardware is capable of performing, so it's an approximate measure of total available computing power.

A high-end consumer-grade CPU can achieve on the order of a few hundred gigaflops (let's say 250, just for a nice round number). https://boinc.bakerlab.org/rosetta/cpu_list.php

A petaflop is therefore about four thousand of those; multiply by another thousand to get an exaflop.

For another point of comparison, a high-end GPU might be on the order of 40-80 teraflops. https://www.tomshardware.com/reviews/gpu-hierarchy,4388-2.ht...

[metadat]

How many teraflops in an exaflop? The tera is screwing me up.. Google not helping today, so many cards.

[aaronblohowiak]

https://en.m.wikipedia.org/wiki/Metric_prefix

[bryanlarsen]

Anybody spending tens of billions annually on Nvidia hardware is going to be willing to spend millions to port their software away from CUDA.

[pjmlp]

First they need to support everything that CUDA is capable of in programing language portfolio, tooling and libraries.

[bryanlarsen]

A typical LLM might use about 0.1% of CUDA. That's all that would have to be ported to get that LLM to work.

[pjmlp]

Which is missing the point why CUDA has won.

Then again, maybe the goal is getting 0.1% of CUDA market share. /s

[imtringued]

Nvidia has won because their compute drivers don't crash people's systems when they run e.g. Vulkan Compute.

You are mostly listing irrelevant nice to have things that aren't deal breakers. AMD's consumer GPUs have a long history of being abandoned a year or two after release.

[pjmlp]

CUDA C++, CUDA Fortran, CUDA Anything PTX, plus libraries, IDE integration, GPU graphical debugging, aren't only nice to have things.

[its_down_again]

In the words of Gilfoyle-- I'll bite. Why has CUDA won?

[pjmlp]

CUDA C++, CUDA Fortran, CUDA Anything PTX, plus libraries, IDE integration, GPU graphical debugging.

Coupled with Khronos, Intel, AMD never delivering anything comparable with OpenCL, Apple losing interest after Khronos didn't took OpenCL into the direction they wanted, Google never adopting it favouring their Renderscript dialect.

[echelon]

For the average non-FAANG company, there's nothing to port to yet. We don't all have the luxury of custom TPUs.

[talldayo]

To slower hardware? What are they supposed to port to, ASICs?

[adgjlsfhk1]

if the hardware is 30% slower and 2x cheaper, that's a pretty great deal.

[selectodude]

Power density tends to be the limiting factor for this stuff, not money. If it's 30 percent slower per watt, it's useless.

[Wytwwww]

The ratio between power usage and GPU cost is very, very different than with CPUs, though. If you could save e.g. 20-30% of the purchase price that might make it worth it.

e.g. you could run a H100 at 100% utilization 24/7 for 1 years at $0.4 per kWh (so assuming significant overhead for infrastructure etc.) and that would only cost ~10% of the purchase price of the GPU itself.

[wbl]

Power usage cost isn't the money but the capacity and cooling.

[sangnoir]

CUDA is the assembly to Torch's high-level language; for most, it's a very good intermediary, but an intermediary nonetheless, as it is between the actual code they are interested in, and the hardware that runs it.

Most customers care about cost-effectiveness more than best-in-class raw-performance, a fact that AMD has ruthlessly exploited over the past 8 years. It helps that AMD products are occasionally both.

[pjmlp]

CUDA is much more than that, and missing that out is exactly why NVidia keeps winning.

[imtringued]

Again, I have AMD hardware and can't use it.

[pjmlp]

AMD is to blame for where they stand.

[jillesvangurp]

Software will bridge the gap. There are simply too many competing platforms out there that are not Nvidia based. Most decent AI libraries and frameworks already need to support more than just Nvidia. There's a reason macs are popular with AI researchers: many of these platforms support Apple's chips already and they perform pretty well. Anything that doesn't support those chips, is a problem waiting to be fixed with plenty of people working on fixing that. If it can be fixed for Apple's chips, it can also be fixed for other people's chips.

And of course there is some serious amount of money sloshing around in this space. Things being hard doesn't mean it's impossible. And there's no shortage of extremely well funded companies working on this stuff. All your favorite trillion $ companies basically. And most of them have their own AI chips too. And probably some reservations about perpetually handing a lot of their cash to Nvidia.

If you want an example of a company that used to have a gigantic moat that is now dealing with a lot of competition, look at Intel. X86 used to be that moat. And that's looking pretty weak lately. One reason that AMD is in the news a lot lately is that they are growing at Intel's expense. Nvidia might be their next target.

[NineStarPoint]

A high grade consumer gpu a (a 4090) is about 80 teraflops. So rounding up to 100, an exaflop is about 10,000 consumer grade cards worth of compute, and a petaflop is about 10.

Which doesn’t help with understanding how much more impressive these are than the last clusters, but does to me at least put the amount of compute these clusters have into focus.

[vitus]

You're off by three orders of magnitude.

My point of reference is that back in undergrad (~10-15 years ago), I recall a class assignment where we had to optimize matrix multiplication on a CPU; typical good parallel implementations achieved about 100-130 gigaflops (on a... Nehalem or Westmere Xeon, I think?).

[NineStarPoint]

You are 100% correct, I lost a full prefix of performance there. Edited my message.

Which does make the clusters a fair bit less impressive, but also a lot more sensibly sized.

[winwang]

4090 tensor performance (FP8): 660 teraflops, 1320 "with sparsity" (i.e. max theoretical with zeroes in the right places).

https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid...

But at these levels of compute, the memory/interconnect bandwidth becomes the bottleneck.

[stonemetal12]

According to Wikipedia the previous #1 was from 2022 with a peak petaflops of 2,055. This system is rated at 2,746. So about 33% faster than the old #1.

Also, of the top 10, AMD has 5 systems.

https://en.wikipedia.org/wiki/TOP500

[smokel]

> P.s. how much is an exaflop or petaflop

1 petaflop = 10^15 flops = 1,000,000,000,000,000 flops.

1 exaflop = 10^18 flops = 1,000,000,000,000,000,000 flops.

Note that these are simply powers of 10, not powers of 2, which are used for storage for example.

[ok123456]

People have been chipping away at this for a while. HIP allows source-level translation, and libraries like Jax provide a HIP version.

[shmerl]

There is ZLUDA to break the lock-in for those who are stuck with it. The rest will use something else.

[fweimer]

Isn't porting software to the next generation supercomputer pretty standard for HPC?

[LeanderK]

its possible. Just look at Apples GPU, its mostly supported by torch, what's left are mostly edge-cases. Apple should make a datacenter GPU :D that would be insanely funny. It's actually somewhat well positioned as, due to the MacBooks, the support is already there. I assume here that most things translate to linux, as I don't think you can sell MacOS in the cloud :D

I know a lot developing on apples silicon and just pushing it to clusters for bigger runs. So why not run it on an apple GPU there?

[Wytwwww]

> Apple should make a datacenter GPU

Aren't their GPUs pretty slow, though? Not even remotely close to Nvidia's consumer GPU with only (significant) upside being the much higher memory capacity.

[talldayo]

> what's left are mostly edge-cases.

For everything that isn't machine learning, I frankly feel like it's the other way around. Apple's "solution" to these edge cases is telling people to write compute shaders that you could write in Vulkan or DirectX instead. What sets CUDA apart is an integration with a complex acceleration pipeline that Apple gave up trying to replicate years ago.

When cryptocurrency mining was king-for-a-day, everyone rushed out to buy Nvidia hardware because it supported accelerated crypto well from the start. The same thing happened with the AI and machine learning boom. Apple and AMD were both late to the party and wrongly assumed that NPU hardware would provide a comparable solution. Without a CUDA competitor, Apple would struggle more than AMD to find market fit.

[LeanderK]

well, but machine learning is the major reason we use GPUs in the datacenter (not talking about consumer GPUs here). The others are edge-cases for data-centre applications! Apple is uniquely positioned exactly because it is already solved due to a significant part of the ML-engineers using MacBooks to develop locally.

The code to run these things on apples GPUs exist and is used every day! I don't know anyone using AMD GPUs, but pretty often its nvidia on the cluster and Apple on the laptop. So if nvidia is making these juicy profits, i think apple could seriously think about moving to the cluster if it wants to.

[talldayo]

Software developers using Macbooks doesn't mean Apple solved the ML problem. The past 10 years of MacOS removing features has somewhat proved that software developers will keep using Macs even when the featureset regresses. Like how Apple used to support OpenCL as a CUDA alternative, but gave up on it altogether to focus on simpler, mobile-friendly GPU designs.

The Pytorch MPS patches are a fun appeasement for developers, but they didn't unthrone Nvidia's demand. They didn't beat Nvidia on performance per watt, they didn't match their price, their scale or CUDA's featureset, and they don't even provide basic server drivers. It's got nothing to do with what brand you prefer and everything to do with what makes actual sense in a datacenter. Apple can't take on Nvidia clusters without copying Nvidia's current architecture - Apple Silicon's current architecture is too inefficient to be a serious replacement to Nvidia clusters.

If Apple wanted to have a shot at entering the cluster game, that window of opportunity closed when Apple Silicon converged on simplified GPU designs. The 2w NPUs and compute shaders aren't going to make Nvidia scared, let alone compete with AMD's market share.

[quickthrowman]

> But how will AMD or anyone else push in? CUDA is actually a whole virtualization layer on top of the hardware and isn't easily replicable, Nvidia has been at it for 17 years.

NVidia currently has 80-90% gross margins on their LLM GPUs, that’s all the incentive another company needs to invest money into a CUDA alternative.

[okdood64]

Maybe the DOJ will come in and call it anti-trust shenanigans.

Not that I would want this...