[muxamilian]

> Unlike BBR, Brutal operates on a fixed rate model and does not reduce its speed in response to packet loss or RTT changes. [...] It's particularly effective at seizing bandwidth in congested, best-effort delivery networks, hence its name.

So if there's one TCP flow using Brutal, all other traffic gets pushed out. Fair queuing can prevent this. If one can be sure that there's fair queuing, one can do much smoother congestion control: https://github.com/muxamilian/fair-queuing-aware-congestion-...

[mhandley]

It's interesting how long we've been waiting for fair queuing to fix congestion problems in the Internet. The first proper research on this was published in 1989: https://dl.acm.org/doi/10.1145/75246.75248

I'm a big fan of fair queuing, and have it enabled for my home network. But in core routers, the best approximation is likely to be WFQ, where you're likely to have each flow hashed to one of something like 256 queues. This means one badly behaved flow can't force well-behaved traffic out of the way and take over the whole link, but it can take over its WFQ queue, starving well behaved flows that hash to the same queue.

I'm not aware of any backbone router that implements true fair queuing. But even if all routers did, it's not a complete solution. Typically flows are mapped to queues based on the 5-tuple (src IP, dst IP, src port, dst port, proto). If you do this, then all Brutal-NG needs to do is use many source ports so it gets many queues, thus many times its fair share, and take over the link again. In fact, this would enable DoS attacks on router state, so no-one is going to do this.

An alternative would be to map to queues using just the source and destination IP addresses. But this has problems too. Brutal-NG could spoof the source address of most of the packets (but send ACKs back to the one unspoofed address), again taking over the link. And it could still cause DoS issues on router state.

The only thing you can't spoof if you want to actually exchange data (as opposed to DoSing the network) is the destination IP address. But now one Brutal flow can achieve the same fair share as all the traffic headed for a busy Google server or an entire ISP's CGNAT. Equally, one flow Brutal flow sending to a host behind the CGNAT can deny service to everyone else sending to the same CGNAT IP address.

So in the end, while I really like what fair queuing does for my VoIP latency on my home network, it is unlikely to ever be a complete solution for constraining misbehaving flows.

[dtaht]

We designed cake to be implementable in hardware if anyone could be convinced it was useful: https://blog.cerowrt.org/post/juniper/ - I thought this test series rather compelling.

1) The data paths MUST prepend a timestamp to every packet 2) HW invsqrt for codel or cobalt AQM is 3k gates, checked on output 3) 32k 4 way set associative cache is a std IP block 4) HW Packet hasher - three different outputs

There are a few states in cake that are optional, but I remain boggled that step 1!! did not happen within 2 hardware releases of switch chips after van Jacobson and kathie Nichols published codel as a replacement for RED in 2012.

https://queue.acm.org/detail.cfm?id=2209336

[dtaht]

Nice to see you mark! and that you are using CAKE. Cake has a feature that helps on regulating multiple flows to a single destination: per host/per flow FQ. In the middle of a network this only works well with IPv6, along the edge it has a nat mode that helps there. We put it in to manage things like bittorrent really well.

Example:

https://www.reddit.com/r/eero/comments/u7xm83/gen_2_sqm_vs_g...

The 8 way set associative hash is also much better than the direct map hash other things use, and I agree 256 flows is too small, 1024 set associative is great, 16k direct map comparable.

[xxpor]

Fair queuing would be extremely difficult to support while maintaining overall peak performance. Way too much shared state.

[mhandley]

I can think of ways to approximate FQ on an input-queued switch while maintaining performance, but it doesn't really help because the total number of queues being a function of the traffic still opens you up to resource exhaustion attacks.

[muxamilian]

Fair queuing shouldn't be necessary in the core. If every base station/WiFi router/client device support fair queuing that should be enough.

[dtaht]

I think FQ helps everywhere, and FQ + aqm is needed at every fast->slow transition on the network. Ideally the core is over provisioned... except where it is not.

[dtaht]

Pushing CAKE to 10Gbit/xeon core in libreqos currently (the bottleneck is on the read path actually). Or, 10k ISP subscribers per $2k xeon box.

[xxpor]

10 gbit per xeon core is bad though

[dtaht]

The bottleneck is mostly on the read path,even leveraging xdp heavily. dpdk/vpp would be faster. In the case however of an ISP 10k subscribers/network segment at 25Gbit is more than enough and busies 40% of 20 cores, on libreqos. Others are making boxes capable of more using smarter ethernet cards. If only more ISPs cared about QoE, we could cause a run on ebay on slightly obsolete Xeons and finish up fixing the bufferbloat problem right quick!