[gmazza]

> Wrong: you see 10KB/s download speed because you are not throttling the incoming packets but the outgoing packets!

Yep. TC's default is to policy outgoing traffic, which in OP's example is a bunch of TCP ACKs essentially. Instead, they should be using ingress keyword, something like described here:

http://blog.stevedoria.net/20050906/ingress-policing-with-li...

Caveat emptor: ingress rate-limiting is hard. Long story short, it all boils down to what you do with non-confirming packets: There are two alternatives, and both are rather sub-optimal. You can either buffer/delay packets in kernel space (default, which leads to bufferbload and memory waste), or drop (which author linked above opted for, which leads to excessive retransmits and bandwidth waste).

[wtallis]

The drop vs buffer decision is no harder for outgoing or incoming packets. In either case: if you're trying to simulate a different kind of network, do what that network does. If you're just trying to get good QoS on your gateway router, then use a smart AQM that will buffer only to the extent that is reasonable, and then drop or ECN mark when buffering threatens to add too much latency.

[mseebach]

What actually happens when Internet traffic makes the jump from my ISPs big-pipe backbone connection to my much slower last-mile connection? They clearly can't cache the world, and "excessive retransmits and bandwidth waste" doesn't seem to be the case?

[wtallis]

It's common to find that the buffers in front of the bottleneck last-mile link are sized for the highest tier of service regardless of what speed you're actually subscribed for. DSLReports has the only browser-based speed test that measures bufferbloat, and they have extensive data from that testing: https://www.dslreports.com/speedtest/results/bufferbloat?up=...

Your modem and the box on the other end of that bottleneck link are probably buffering far more than is reasonable. There's simply no reason for a cable modem to ever have in excess of 1s worth of backlog.

[Sunset]

They drop your packets, both for ingoing and outgoing overflow.

[mseebach]

But how is that different from the proposed method of rate limited which causes "excessive retransmits and bandwidth waste"?

Not being pedantic, genuinely trying to understand how this stuff works.

[gmazza]

Middle ground between dropping [1] and buffering is Active Queue Management as wtallis pointed out above. State of the art AQM on Linux nowadays is CoDel [2].

Big network gear vendors that ship to ISPs have adopted early forms of AQM (both RED [3] and proprietary algorithms) quite a while ago - they had to:

(a) backbone routers have much smaller buffer-to-bandwidth ratio (compared to a PC or even home router), so endless buffering is not an option;

(b) buffer tail drops (i.e. what drop keyword does when added to tc filters/disciplines) interact really poorly with TCP bandwidth control algorithms, and rather badly with RTP streams, too (so drop is not an option either - it ruins user's connections; and

(c) ISPs and carriers would typically run links (on average) much closer to saturation than your typical home router, so situation where router has to make buffer/drop decision is much, much more frequent.

[1] My point above was that you don't really want to drop packets on the receiver side, after that packet already traversed expensive part of the network.

[2] https://en.wikipedia.org/wiki/CoDel

[3] https://en.wikipedia.org/wiki/Random_early_detection