[dnautics]

I don't think this is correct. For inference, the bottleneck is memory bandwidth, so if you can hook up an FPGA with better memory, it has an outside shot at beating GPUs, at least in the short term.

I mean, I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago.

[fooblaster]

Show me a single FPGA that can outperform a B200 at matrix multiplication (or even come close) at any usable precision.

B200 can do 10 peta ops at fp8, theoretically.

I do agree memory bandwidth is also a problem for most FPGA setups, but xilinx ships HBM with some skus and they are not competitive at inference as far as I know.

[checker659]

Said GPUs spend half the time just waiting for memory.

[fooblaster]

Yep, but they are still 50x faster than any fpga.

[dnautics]

probably not B200 level but better than you might expect:

https://www.positron.ai/

i believe a B200 is ~3x the H200 at llama-3, so that puts the FPGAs at around 60% the speed of B200s?

[fooblaster]

I wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.

[dnautics]

ok? i worked on those devices, those numbers are real. theres a reason why they compare to h200 and not b200

> I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago