[klohto]

What are you talking about? Why would you multiply the bandwidth? 8 4090s is still 1000GB/s. While the M2 Ultra is 800GB/s with a top of 192GB VRAM. Metal can access ~155GB, so you need a bit more, but your comparison makes absolutely zero sense.

[boroboro4]

There are different ways to run LLMs on multiple GPUs, one of them (called tensor parallelism) in low batch scenarios would be multiplying bandwidth between different GPUs. So no, 8 4090s is not 1000 GB/s.

[behnamoh]

you've heard something and are regurgitating it without fully understanding it.

[boroboro4]

I’m developing inference engine, so I actually do understand how it works. As well as other types of parallelism and how exactly they do different trade offs

[klohto]

let me know how is the PCIe bandwidth treating you

[boroboro4]

Since we’re talking about small batch sizes PCIe bandwidth isn’t as important - intermediate hidden state is magnitude smaller than weights.

[andersa]

When you have 8 GPUs, you can use more than 1 at a time.