[fooblaster]

Show me a single FPGA that can outperform a B200 at matrix multiplication (or even come close) at any usable precision.

B200 can do 10 peta ops at fp8, theoretically.

I do agree memory bandwidth is also a problem for most FPGA setups, but xilinx ships HBM with some skus and they are not competitive at inference as far as I know.

[checker659]

Said GPUs spend half the time just waiting for memory.

[fooblaster]

Yep, but they are still 50x faster than any fpga.

[dnautics]

probably not B200 level but better than you might expect:

https://www.positron.ai/

i believe a B200 is ~3x the H200 at llama-3, so that puts the FPGAs at around 60% the speed of B200s?

[fooblaster]

I wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.

[dnautics]

ok? i worked on those devices, those numbers are real. theres a reason why they compare to h200 and not b200

> I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago

[fooblaster]

I'd like to know more. I expect these systems are 8xvh1782. Is that true? What's the theoretical math throughput - my expectation is that it isn't very high per chip. How is performance in the prefill stage when inference is actually math limited?