[pipsterwo]

1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above)

Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...

[boroboro4]

The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.

[pipsterwo]

Totally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context