[klohto]

let me know how is the PCIe bandwidth treating you

[boroboro4]

Since we’re talking about small batch sizes PCIe bandwidth isn’t as important - intermediate hidden state is magnitude smaller than weights.