[Wytwwww]

> wouldn’t perf per dollar look completely different

Isn't the cost of actually running the card dwarfed by the cost of the card itself? e.g. even if we look at an old card like the A100 at 100% 24/7 that's about 3500 kWh per year which is what $300-$500 depending on the location?

if we Intel could come out with a card that could do anything the A100 does at half the price but 4x power usage it could still be somewhat competitive.

[maxwell22]

That’d depend on how many of these cards you need to get the same perf as an A100. Most of the results being shown are batch 1 (use the cards to serve 1 user). But in practice you’d use a single A100 to serve thousands of users concurrently (you can’t do that with the Gaudi 2 though).

The article measures that Gaudi 2 is competitive at “latency for batch size 1”, but that isn’t really a metric anyone cares about.

So Intel would need to sell these for much less than half, and then perf per watt would need to be much better (the article measures that it is currently worse than A100).

Comparing these things is hard, “if Intel were to sell these” is already speculation, since they aren’t on sale.

The article is right that perf per dollar is better, but that’s only because Intel is not making money out of these. As a user that’s a red flag, because if that continues these will be discontinued, and then any investment I do right now in supporting these in my SW stack goes to waste.