[yubblegum]

I just had to double check (have not been paying attention for a couple of years) but indeed it seems GPU underutilization remains a fact and the numbers are pretty significant. Main issues are being memory bound so the compute sits idle.

[ninkendo]

The actual computation speed isn't as important nowadays but it doesn't really change the conclusion with respect to whether they're underutilized.

Because the main reason for the price premiums in AI-class GPUs are the gobs of insanely fast memory, and that is very much not underutilized. AI companies grab GPUs with as much memory (at the fastest memory bandwidth) as possible and underclock the GPU to save on power. Linus Tech Tips had a great video about the H200 that touched on this this week: https://www.youtube.com/watch?v=lNumJwHpXIA

[davedx]

Tasks being memory bound is not the same thing as GPU's being idle for economic reasons though.