[noosphr]

So does llm inference. You're lucky if you hit 40% of the advertised flops.

[fwipsy]

Right, but datacenter GPUs optimized for LLM training/inference would have a bandwidth:compute ratio scaled to that workload.

[noosphr]

No they don't.

[dapperdrake]

Then where are the HDMI ports on Nvidia's current data center GPU product lines?

[eru]

Why not? Seems like they would be poorly optimised?

[noosphr]

Because llm inference is not the only workload a GPU can do and custom silicon cost $10b a chip.

[eru]

AI optimised cards is the biggest cashcow for nvidia. They are definitely doing custom silicon. And Google et al have cards that don't even pretend to be able to do graphics.

[noosphr]

There is more than one workload in AI. Inference for llms is memory constrained on even a single card. Training for llms is memory constrained on the level of racks. In both cases you hardly ever see more than 40% of advertised flops used.