[mehrdadn]

Makes sense if this happens, but does this actually happen to you? I've heard vague and rather dubious third-hand stories along these lines, but I've never actually encountered a router that needs rebooting to keep working well.

This actually seems bizarre to me now that I think more about it. The routers I've seen allow something like a few hundred thousand established connections over like a ~week. Say 300,000 over 3 days. To exhaust this you'd need to establish on average one new connection every single second (300000/3/24/60/60 ≈ 1), continuously for a week, while also timing out on every single one of them silently. Surely a normal person wouldn't exhaust such a table?

[dijit]

Exhausted NAT state tables is excessively common, evictions happen silently and the assertion that a reboot is required is for other reasons which I think are likely unrelated.

Professionally I run one (two, actually) of those annoying 'always online video games' and state drops in low quality routers is the most common cause of VOIP drop.

It seems like most router firmware has some kind of intelligent sensing software to see if there's a lot of traffic going over a state and then attempting to avoid evicting it. But for VOIP which can sometimes be silent.. or for a person not moving around in a game (and thus sending/recieving very few and very tiny updates) it can be seen.

Now; you want concrete evidence, and unfortunately the kinds of routers most people have (Say, a Virgin Hub 3.0 which is based on the Touchstone TG2492[0]) does not lend itself to being monitored well.

We're in some luck though, as I happen to run something immeasurably more powerful: a PfSense branded NetGate APU2[1]

PfSense absolutely /loves/ letting you know how it feels; and if we assume that I'm a "normal" user, (I have 1 laptop, 1 phone and an apple watch as the only devices on my network right now and I'm just browsing like normal) then we have some measure of how much memory a state table really consumes.

My state table currently contains a mere 170 states (according to iftop), but it's not really hurting my memory:

> 6% of 4030 MiB

Yet, I can see that some states have been forcefully closed, despite having lots of ram available to store too (these statistics were reset yesterday):

   state-mismatch                       748            0.0/s

In general the state table is very busy:

  State Table                          Total             Rate
    current entries                      152               
    searches                        90040931          338.1/s
    inserts                           437333            1.6/s
    removals                          437181            1.6/s

it's worth noting that this device is forcefully configuring itself to hit a max of 403000 states total:

  states        hard limit   403000

So it's not "memory" like you suggest, but since doing nat translation on every single packet is CPU intensive, states can be dropped if the table can't keep up.

[0]: 256MB of ram reserved for the state table it seems: https://deviwiki.com/wiki/Virgin_Media_Super_Hub_3

[1]: 4G of general purpose ram: https://www.firewallhardware.it/en/apu2-3nic/

[mehrdadn]

Thanks for sharing. While I have a hard time grasping your usage (why in the world are 3 devices opening 1.6 connections every second?), it's not really relevant as your own data shows state tables don't get exhausted, right? Your table only has 152 entries, which is quite a far cry from exhausting its 403,000 slots.

[dylz]

This is fairly normal and common, especially if you browse without aggressive ad blocking.

I routinely see a single ad impression make over 20-50 connections outbound, and repeatedly close and reopen or randomly open new ones for various reasons, the most common being some form of "anti ad fraud" tracking that repeatedly polls to get an average or median latency, new connections and requests firing on every mouse move, etc.

Would also be entirely unsurprised if phones that had free mobile games and equivalent were polling and sending stuff like location data every minute.

[dijit]

My point is that even when I don't quite run out, something is dropping states.

the Hard limit is just one imposed by the OS, it doesn't seem to matter that I have absurd amounts of free memory, or that the kernel is quite content with loading up hundreds of thousands of states: they still get dropped.

And like I said, my hardware and software platform is many dozens of times more advanced than what most people are using at home.

As for the usage; easily explained by: every single website I open, all of the things that website asks my browser to pull in, every DNS request, every NTP update and every 'ping' to see if the device is online-- counts as a new state.

[mehrdadn]

You're talking about the state-mismatch rate being nonzero, right? I take it as a given that that represents the router dropping states? And you're assuming that must be due to NAT slot exhaustion? If that's what you're saying, it clearly doesn't square with the 152 slots being in use currently (nor does it make sense to me otherwise, given everything I explained above). So either the states are being dropped due to a different reason than you're claiming (I see no link to table exhaustion? it seems like a conjecture), or I'm completely missing a giant piece of the puzzle. Heck, if I take the name at face value, "state mismatch" just sounds like it could be due to a bug in the connection endpoints (or random package spamming from the internet...), rather than anything related to the router at all.

[dijit]

Routers, especially cheap ones, are often equipped with weak CPUs because they aren’t designed to handle heavy processing loads. It’s not like you’re calculating physics or processing 3D animation directly on your router, right?

But network address translation _can_ be a processing-heavy task.

Every single packet that leaves the private network needs to be translated, and every single packet that comes in from the public network needs to be translated. Each individual translation may be simple enough, but with heavy internet use, it all adds up.

Here’s my network activity while browsing the web: https://i.imgur.com/oP8PrX4.png, with one 720p YouTube video open in a tab and a dozen other tabs for various websites, all in the Edge browser.

The top nine processes are using an average of 1,182,149 bytes per second. Every network interface has a maximum transmission unit (MTU), which is the largest size that a data packet can be. Ethernet and Wi-Fi have an MTU of 1,500 bytes.

My computer, doing nothing more than watching a YouTube video, is putting a minimum load on my router of 788 packets per second. That’s assuming the bytes are all divided into 1,500-byte packets, which isn’t the case in real world usage. Somewhere between 1,000 to 3,000 packets per second is more realistic.

The load is worse during bandwidth-intensive activities, such as multiplayer gaming and torrenting. In fact, torrenting is so intensive that it’s the primary cause of NAT issues for home users today. (Open connections to dozens/hundreds of peers, with each connection involving high-speed downloads and uploads.)

And it’s not just one computer on a private network. It is commont to have a smartphone or two, maybe a tablet, smart TV, plus a handful of other devices for the rest of the people sharing the living space. They all need network address translations too!

At the end of the day, we’re talking thousands and thousands of data packets per second, all translated by a weak CPU that can’t keep up. It’s one reason why cheap routers are prone to slowing down.

Notably: while doing that (and opening youtube) my state table grew to just under 400 states. So, youtube needs a lot of connections it seems.

[mehrdadn]

I'm sorry but I still don't get how any of this implies NAT table exhaustion. A few hundred entries is literally 3 orders of magnitude away from a few hundred thousand entries. I don't see the problem.