[Plutoberth]

This board has four 2.5G ports, not four 10G ports, so the 16G bandwidth for four PCie 2.0 lanes is more than enough.

[mlyle]

Right... the parent is saying that 4x10G is pointless compared to 4x2.5G, because PCIe 4 lanes will top out at forwarding around 7gbps of traffic.

You can't do line rate on all ports either (limited by PCIe alone, let alone CPU for smaller packets), but you can certainly fill an individual port, which I suspect is the goal.

[q3k]

> because PCIe 4 lanes will top out at forwarding around 7gbps of traffic [...] limited by PCIe alone

Are you sure about that? With 5GT/s (or 500MB/s) per lane, and with 4 lanes, that should be plenty, no? Intel adapters like the x520-DA2 are specced at 2x 10G, and use PCIe 2.0 x8.

FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.

[adrian_b]

The so-called 5 Gb/s PCIe 2.0 lanes are in fact 4 Gb/s lanes.

Ethernet speeds are given in data bits per second, while USB 3.0, PCIe 1.0 and PCIe 2.0 add in their speed values the extra bits used for encoding.

On the other hand, 10 Gb/s USB and 8 Gb/s PCIe 3.0 have really those speeds. Complicated :-(

One PCIe 2.0 lane is more than enough for 2.5 Gb/s Ethernet, but not enough for 5 Gb/s Ethernet or 10 Gb/s Ethernet.

[mlyle]

> FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.

Yah, I said forwarding-- so each packet goes over the PCIe link twice.

So, to use your test-- 3.7 x 4 / 2 = 7.4gbps. I said around 7. 7.4gbps would be a bit more than I'd expect, but I'd not be shocked.

[q3k]

But PCIe is full duplex! With PCIe 2.0 x4 there's 4 lanes in each direction [1], so when 'forwarding' over a single 10G link you can expect to send and receive simultaneously at the speed I mentioned earlier.

[1] - https://en.wikipedia.org/wiki/PCI_Express#Pinout

[mlyle]

Yah, I guess dividing by 2 isn't fair. But transmitting does impact receiving and vice-versa: when you're reading DMA descriptors, you need to wait/hold for posted completions, etc. It's not fully uncontended between send and receive, but more uncontended than a naive division by 2 would imply.