[modeless]

Intel Gaudi 3 has more interconnect bandwidth than this has memory bandwidth. By a lot. I guess they can't be fairly compared without knowing the TCO for each. I know in the past Google's TPU per-chip specs lagged Nvidia but the much lower TCO made them a slam dunk for Google's inference workloads. But this seems pretty far behind the state of the art. No FP8 either.

[leetharris]

They are different architectures optimized for different things.

From the Meta post: "This chip’s architecture is fundamentally focused on providing the right balance of compute, memory bandwidth, and memory capacity for serving ranking and recommendation models."

Optimizing for ranking/recommendation models is very different from general purpose training/inference.

[janalsncm]

Translation: you don’t need to serve 96 layer transformers for ranking and recommendation. You’re probably using a neural net with around 10-20 million parameters. But it needs to be fast and highly parallelizable, and perhaps perform well in lower precisions like f16. And it would be great to have a very large vector LUT on the same chip.

[samspenc]

Is there a better way to compare performance across these high-end chips? The only comparable numbers I was able to find were the TFLOPS.

Meta seems to be reported these numbers for this v2 chip:

    708 TFLOPS/s (INT8) (sparsity)
    354 TFLOPS/s (INT8)

And I see Nvidia reporting these numbers for its latest Blackwell chips https://www.anandtech.com/show/21310/nvidia-blackwell-archit...

    4500 T(FL)OPS INT8/FP8 Tensor 

Am I understanding correctly that Nvidia's upcoming Blackwell chips are 5-10x faster than this one Meta just announced?

[ipsum2]

To a rough approximation, yes. The blackwell chip is also ~10x larger in surface area than MTIA, so the costs are proportional.

[modeless]

Yeah, it may fit their current workload perfectly, but it doesn't seem very future proof with the limited bandwidth. Given how fast ML is evolving these days I question if it makes sense to design and deploy a chip like this. I guess they do have a very large workload that will benefit immediately.

[tony_cannistra]

Don't mean to single you out at all, but I find this comment to be a great example of how the "ML Hype" is perceived by a certain segment folks in our industry.

The development of this chip shows that it doesn't (and shouldn't!) matter to the ML teams at Meta how 'fast ML is evolving.'

Indeed what it demonstrates is that a huge, global, trillion-dollar business has operationalized an existing ML technology to the extent that they can invest into, and deploy, customized hardware for solving a business problem.

How ML "evolves" is irrelevant. They have a system which solves their problem, and they're investing in it.

[airstrike]

Not to mention the capabilities they developed by actually creating this and what they'll be able to do next thanks to this experience.

You've gotta learn to walk before you can run

[janalsncm]

In their defense, it’s because the article is (understandably) sparse on details about what makes the requirements of their ranking models different from image classification or LLMs. Unless you work in industry it’s unlikely you will have heard of DeepFM or ESMM or whatever Meta is using.

And building out specialized hardware does lock you in to a certain extent. Want to use more than 128GB of memory? Too bad, your $10B chip doesn’t support that.

[sangnoir]

> Want to use more than 128GB of memory? Too bad, your $10B chip doesn’t support that.

Which is probably why Meta is also buying the biggest Nvidia datacenter cards by the shipload. There is no need to run inference for a small model - say for a text-ad recommendation system - on an H100 with attendant electricity and cooling costs.

[namibj]

Also, like, FP tensor cores are way more expensive than fixed-point tensor cores, and with some care, it's very much practical to even train DNNs on them.

E.g. it's common to have a full-width accumulator and e.g. s16 gradients with u8 activations and s8 weights, with the FMA (MAC) chain of the tensor multiply operation post-scaled with a learned u32 factor plus follow-up "learned" notify, which effectively acts as a fixed-point factor with learned position of it's point, to re-scale the outcome to the u8 activation range.

By having the gradients by sufficiently wider, it's practical to use a straight-through estimator for backpropagation. I read a paper (kinda two, actually) a few months ago that dealt with this (IIRC one of them was more about the hardware/ASIC aspects of fixed-point tensor cores, the other more about model training experiments with existing low precision integer-MAC chips IIRC particularly for interference in mind). If requested, I can probably find it by digging through my system(s); I would have already linked it/them if the cursory search hadn't failed.

[prpl]

To me, it’s bizarre to see the HPC mindset taking hold again after the cloud/commodity mindset dominated the last 16 years.

You don’t always need a Ferrari to go to the store

[thorncorona]

WDYM by HPC mindset?

[rfoo]

"The only meaningful benchmark in the world is LAPACK and only larger than ever monolithic problem instances matter, I don't know what you're talking about, 'embarrassingly parallel'? What a silly word! Serving web requests concurrently? Good for you, congratulations, but can you do parallel programming?"

Sorry if this make anyone feels bad. It certainly made myself uncomfortable typing it out though.

[Aurornis]

> Yeah, it may fit their current workload perfectly, but it doesn't seem very future proof

It’s custom silicon designed for a specific, known workload. It’s not designed to be a general purpose part or to be future proofed for unknown future applications.

When a new application comes along with new requirements, the teams will use their experience to create a new chip targeting that new application.

That’s the great part about custom silicon: You’re not hitting general specs for general applications that you may not even know about yet. You’re building one very specific thing to do a very specific job and do it very well.

[noiseinvacuum]

Right and they have a LOT of GPUs from Nvidia for handle the unknown. Custom silicon for custom workloads seems like a good strategy specially considering the capabilities that the team will develop along the way.

[giantrobot]

Offloading a known workload to a custom chip can also save a lot on operations costs, particularly power. Facebook is interested in workload operations per watt rather than raw floating point operations per watt. A GPU might have better raw specs but if the whole GPU package has worse workload ops per watt, a custom chip is likely better.

At Facebook's scale the spherical cow raw performance stats don't matter nearly as much as real world workloads per ops dollar. They can also repurpose their GPUs to other workloads and let their custom chips handle the boring baseline stuff.

[chabons]

> Intel Gaudi 3 has more interconnect bandwidth than this has memory bandwidth.

LPDDR5 vs HBMe2. I'm guessing there's a 2-5x price difference between those, but even so it's an interesting choice, I don't know any other accelerators which spec DDR. But yeah, without exact TCO numbers it's hard to compare exactly.

[namibj]

Bandwidth is far more power hungry for DDR, but capacity is far cheaper.

If the bandwidth capability of DDR suffices, HBM isn't worth it.

At least with LPDDR's; GDDRs may well not be worth it under data center TCO considerations due to the high interface power usage. Feel free to correct me if I'm mistaken, the numbers in question aren't too easy to search for so I didn't confirm this (LPDDR vs. GDDR) part.

[chessgecko]

Also its at 90 watts vs 900 watts for gaudi 3, the flops/mem bw per watt is much more comparable.

[modeless]

With high end chips like that it's often possible to get dramatically better efficiency by running it at less than peak power consumption, like 90% performance at 50% power or something like that. It's hard to compare the numbers in a fair way.

[moffkalast]

It would be interesting if this could be made into a reasonably priced (lmao) card for home inference if they intend to mass produce it.

Can't imagine any other reason other than cost as to why they went with LPDDR5, LPDDR5X has more bandwidth and GDDR6 has even more.

[chessgecko]

they didn't use GDDR cause they wanted the memory capacity which is really important for recommendation models. But I totally agree that this is a sort of perfect cost/perf per watt point for a home setup. I really hope they do it, if not for this one at least for v3.

[cma]

Only 48MB of SRAM on Gaudi 3 per die (96 MB across both) vs 256MB here maybe increases the memory bandwidth needs for Gaudi. Way different power consumption too.