[q3k]

> because PCIe 4 lanes will top out at forwarding around 7gbps of traffic [...] limited by PCIe alone

Are you sure about that? With 5GT/s (or 500MB/s) per lane, and with 4 lanes, that should be plenty, no? Intel adapters like the x520-DA2 are specced at 2x 10G, and use PCIe 2.0 x8.

FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.

[adrian_b]

The so-called 5 Gb/s PCIe 2.0 lanes are in fact 4 Gb/s lanes.

Ethernet speeds are given in data bits per second, while USB 3.0, PCIe 1.0 and PCIe 2.0 add in their speed values the extra bits used for encoding.

On the other hand, 10 Gb/s USB and 8 Gb/s PCIe 3.0 have really those speeds. Complicated :-(

One PCIe 2.0 lane is more than enough for 2.5 Gb/s Ethernet, but not enough for 5 Gb/s Ethernet or 10 Gb/s Ethernet.

[mlyle]

> FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.

Yah, I said forwarding-- so each packet goes over the PCIe link twice.

So, to use your test-- 3.7 x 4 / 2 = 7.4gbps. I said around 7. 7.4gbps would be a bit more than I'd expect, but I'd not be shocked.

[q3k]

But PCIe is full duplex! With PCIe 2.0 x4 there's 4 lanes in each direction [1], so when 'forwarding' over a single 10G link you can expect to send and receive simultaneously at the speed I mentioned earlier.

[1] - https://en.wikipedia.org/wiki/PCI_Express#Pinout

[mlyle]

Yah, I guess dividing by 2 isn't fair. But transmitting does impact receiving and vice-versa: when you're reading DMA descriptors, you need to wait/hold for posted completions, etc. It's not fully uncontended between send and receive, but more uncontended than a naive division by 2 would imply.