[boroboro4]

There are different ways to run LLMs on multiple GPUs, one of them (called tensor parallelism) in low batch scenarios would be multiplying bandwidth between different GPUs. So no, 8 4090s is not 1000 GB/s.

[behnamoh]

you've heard something and are regurgitating it without fully understanding it.

[boroboro4]

I’m developing inference engine, so I actually do understand how it works. As well as other types of parallelism and how exactly they do different trade offs

[klohto]

let me know how is the PCIe bandwidth treating you

[boroboro4]

Since we’re talking about small batch sizes PCIe bandwidth isn’t as important - intermediate hidden state is magnitude smaller than weights.