[maxwell22]

There I read that Intel Gaudi is behind all NVIDIA products in all metrics except performance per dollar.

Since one can only access these in the Intel cloud, where Intel has essentially made the few of these that exist freely available, isn’t perf per dollar kind of a joke metric? I mean, if they are free to use, they have infinite perf per dollar. But if Intel were to sell them, you were to buy them, and then you were to pay for their power, then wouldn’t perf per dollar look completely different ?

The article seems to try really hard to avoid saying how much perf per watt these have and how expensive they are. Reads a bit like a marketing piece as part of some Intel / data bricks collab (all companies and vendors do these).

[Wytwwww]

> wouldn’t perf per dollar look completely different

Isn't the cost of actually running the card dwarfed by the cost of the card itself? e.g. even if we look at an old card like the A100 at 100% 24/7 that's about 3500 kWh per year which is what $300-$500 depending on the location?

if we Intel could come out with a card that could do anything the A100 does at half the price but 4x power usage it could still be somewhat competitive.

[maxwell22]

That’d depend on how many of these cards you need to get the same perf as an A100. Most of the results being shown are batch 1 (use the cards to serve 1 user). But in practice you’d use a single A100 to serve thousands of users concurrently (you can’t do that with the Gaudi 2 though).

The article measures that Gaudi 2 is competitive at “latency for batch size 1”, but that isn’t really a metric anyone cares about.

So Intel would need to sell these for much less than half, and then perf per watt would need to be much better (the article measures that it is currently worse than A100).

Comparing these things is hard, “if Intel were to sell these” is already speculation, since they aren’t on sale.

The article is right that perf per dollar is better, but that’s only because Intel is not making money out of these. As a user that’s a red flag, because if that continues these will be discontinued, and then any investment I do right now in supporting these in my SW stack goes to waste.