Hacker News new | ask | show | jobs
by mlyle 1985 days ago
Right... the parent is saying that 4x10G is pointless compared to 4x2.5G, because PCIe 4 lanes will top out at forwarding around 7gbps of traffic.

You can't do line rate on all ports either (limited by PCIe alone, let alone CPU for smaller packets), but you can certainly fill an individual port, which I suspect is the goal.
1 comments
q3k 1985 days ago
> because PCIe 4 lanes will top out at forwarding around 7gbps of traffic [...] limited by PCIe alone

Are you sure about that? With 5GT/s (or 500MB/s) per lane, and with 4 lanes, that should be plenty, no? Intel adapters like the x520-DA2 are specced at 2x 10G, and use PCIe 2.0 x8.

FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.

link
adrian_b 1985 days ago
The so-called 5 Gb/s PCIe 2.0 lanes are in fact 4 Gb/s lanes.

Ethernet speeds are given in data bits per second, while USB 3.0, PCIe 1.0 and PCIe 2.0 add in their speed values the extra bits used for encoding.

On the other hand, 10 Gb/s USB and 8 Gb/s PCIe 3.0 have really those speeds. Complicated :-(

One PCIe 2.0 lane is more than enough for 2.5 Gb/s Ethernet, but not enough for 5 Gb/s Ethernet or 10 Gb/s Ethernet.

link
mlyle 1985 days ago
> FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.

Yah, I said forwarding-- so each packet goes over the PCIe link twice.

So, to use your test-- 3.7 x 4 / 2 = 7.4gbps. I said around 7. 7.4gbps would be a bit more than I'd expect, but I'd not be shocked.

link
q3k 1985 days ago
But PCIe is full duplex! With PCIe 2.0 x4 there's 4 lanes in each direction [1], so when 'forwarding' over a single 10G link you can expect to send and receive simultaneously at the speed I mentioned earlier.

[1] - https://en.wikipedia.org/wiki/PCI_Express#Pinout

link
mlyle 1985 days ago
Yah, I guess dividing by 2 isn't fair. But transmitting does impact receiving and vice-versa: when you're reading DMA descriptors, you need to wait/hold for posted completions, etc. It's not fully uncontended between send and receive, but more uncontended than a naive division by 2 would imply.
link