[throwaway894345]

GPGPU? General Purpose GPU? If embarrassingly parallel CPU algorithms weren't offloaded to the GPU previously, why would the A100/H100 price drop make a difference? We had cheap GPU in the past and we still left plenty of performance on the table with CPU programs because they were easier to build.

Is the idea that previously maintaining GPU programs was expensive whereas now AI makes it cheap? If so, I could buy that line of reasoning.

Maybe relatedly, I expect (hope) the hardware manufacturers will ramp up supply in the meanwhile which would also put downward pressure on GPUs. Right now though this hardware crunch is making me sad, not even because of GPUs but also because of general memory / disk.

[christina97]

GPGPU programming has become significantly easier now and the payoff is bigger (better hardware), due to the immense investment in this due to ML/AI.

[NuclearPM]

What are the best tools for this?

[girvo]

We've not had cheap GPUs with this much VRAM before, though. Might be an interesting change, though I also doubt it personally.

[nl]

As noted in my other comment you can get obsolete GPUs (P100s, BC250s) with lots of RAM on AliExpress now. It hasn't proven revolutionary.

[swiftcoder]

is 16Gb "lots of RAM" in an LLM world? How many of these would you need to stack on a motherboard to inference a decent size model?

[oblio]

At least 4, probably 6+.

[fc417fc802]

BC250 shares memory with system. 16 GB GPUs have long been available to consumers for a small premium. What's never been readily available before is 40+ GB of HBM.

[girvo]

Those don't have a lot of RAM though. Not like these 40, 80GB ones.