Hacker News new | ask | show | jobs
by danceparty 2378 days ago
We are not struggling to do line rate 10G in the kernel. Modern 100Gbe nics (mellanox, solarflare) will happily do line rate with stock upstream kernel for a while now (definitely since 4.x) you only need to tune your irq balancing, and you can probably get away with not even doing that. If you are buying 100gbe nics you are also buying server class (xeon, rome) processors that can keep up.

Source: I operate a CDN with thousands of 100Gbe nics with a stock upstream LTS kernel, and minimal kernel tuning.
1 comments
ra1n85 2378 days ago
You're saying you can forward 100Gbps at line rate (148MPPS) through a stock kernel?
link
danceparty 2378 days ago
You can get within a few percentage points, yes

I just tested this with two hosts with 4.14.127 upstream kernel and upstream mlx5 driver, and mellanox connectx-5 card. Using 16 iperf threads

[SUM] 0.0-10.0 sec 85.1 Gbits/sec

That's pretty close with no tuning, and well beyond 10gb/s we mentioned earlier

link
GhettoMaestro 2378 days ago
Wrong. You’re like an order of magnitude wrong - rofl there is no fucking way the stock Linux kernel will even do 40Mpps at 64 byte packets. It chokes way before that. This is partly why things like DPDK exist.
link
ra1n85 2378 days ago
16 iperf threads...sending at what packet size? Do you understand the notion of line rate? 85Gbps at 1500B is only 7MPPS, which is half of 10Gbps at line rate.
link
Dylan16807 2378 days ago
Where are you getting your definitions? I have never seen "line rate" used to refer to packets per second.
link
ra1n85 2378 days ago
It implies it. Ethernet at 84B per frame is the smallest you can go, thus the line rate - some examples:

https://events19.linuxfoundation.org/wp-content/uploads/2017...

https://www.redhat.com/en/blog/pushing-limits-kernel-network...

https://kernel-recipes.org/en/2014/ndiv-a-low-overhead-netwo...

link
Dylan16807 2378 days ago
"How do you fill a 100GBps pipe with small packets?"

"achieve 10 Gbps line rate at 60B frames"

"reaching line rate on all packet sizes"

Line rate is just bits per second. You have to add in a qualifier about packet size before you're talking about packets per second.

link
bogomipz 2378 days ago
No, PPS and bandwidth are two distinct metrics. Although there can be a linear relationship between them line rate does not always imply a payload equal or close to the interface MTU. You see this with network vendors and some of the higher end gear. Network vendors always give specs for their gear by quoting both metrics. And there is some gear that that is capable of doing line rates even with small packets, example the Cisco ASR 1000:

"For example, because one of the newest Cisco routers, the Cisco ASR 1000 Series Router, is capable of forwarding packets at up to 16 Mp/s with services enabled, it can support the processing of the equivalent of 10 Gb/s of traffic at line rate, with services, even for small packets."[1]

[1] https://tools.cisco.com/security/center/resources/network_pe...

link
toast0 2378 days ago
Does line rate imply smallest packets? For CDN style use cases, you want to use the whole pipe, and it's going to be mostly larger packets.
link
ra1n85 2378 days ago
Yes.

The point of discussion here is that the Linux kernel struggles to do line rate 10Gbps. This was misinterpreted as "the Linux kernel struggles to do 10Gbps".

link
gonzo 2378 days ago
You’re sending 1500 byte MTU (1538 bytes on the wire) or maybe larger (9000 byte MTU) packets.

1538 bytes is 12,304 bits. 10,000,000,000 bits/sec / 12,304 bits/packet is 812,744 packets per second.

Now try it with 64 byte packets, which are 84 bytes on the wire.

14,880,952 packets per second.

And this is 10gbps.

link