[_9jgl]

The issue is that the current OOM killer doesn't support this usage at all.

To extend the analogy: what do you do if grandma comes and fills your house with stuff? You need space to work, so you go and drop it off at the self storage place, but what if she just keeps filling your house up?

The OOM killer will do absolutely nothing until both your house and the whole self storage place are totally full. By that point, you've spent a huge amount of time just driving to and from self storage, so you haven't had time to do any actual work; it would probably have been better to tell grandma that you don't want any more stuff once she filled up your house for the first time.

[quotemstr]

Well, it doesn't help that when grandma calls and asks whether you have room for more stuff, the Linux kernel responds on your behalf, "Yes, of course I have room. I live in a TARDIS." And you then do all driving to the self-storage facility to maintain the illusion as long as you can. I really don't like overcommit.

Anyway, I agree with you that this behavior is annoying, but I think it ought to be possible to fix it (e.g., with memory cgroups or something like Android's lmkd) without giving up on the idea of spilling infrequently-accessed private dirty pages to disk.

[MaxBarraclough]

The analogy is now getting in the way, rather than helping to clarify.

[cdown]

One problem with relying on the OOM killer in general is that the OOM killer is only invoked in moments of extreme starvation of memory. We really have no ability currently in Linux (or basically any operating system using overcommit) to determine when we're truly "out of memory", so the main metric used is our success or failure to reclaim enough pages to meet new demands.

As for the analogy -- there are metrics you can use today to bat away grandma before she starts hoarding too much. We have metrics for how much grandma is putting in the house (memory.stat), at what rate we kick our own stuff out of the house just to appease grandma, but then we realise we removed stuff we actually need (memory.stat -> workingset_refault), and similar. Using this and Johannes' recent work on memdelay (see https://patchwork.kernel.org/patch/10027103/ for some recent discussion), it's possible to see memory pressure before it actually impacts the system and drives things into swap.

[tmyklebu]

> One problem with relying on the OOM killer in general is that the OOM killer is only invoked in moments of extreme starvation of memory. We really have no ability currently in Linux (or basically any operating system using overcommit) to determine when we're truly "out of memory", so the main metric used is our success or failure to reclaim enough pages to meet new demands.

The problem with relying on swap instead of the OOM killer is that, instead of the OOM killer, the user gets invoked in moments of extreme starvation of memory and the whole machine gets rebooted. The OOM killer is far gentler; it only kills processes until the extreme starvation is resolved.

[vardump]

Well, just don't allow overcommit, problem solved.

[quotemstr]

Disallowing overcommit still doesn't solve the whole problem: you can just burn all of RAM and all of swap in commit charge, then swap. Another failure mode is excessive paging IO causing the kernel to spill private memory to the swap file prematurely, preferring instead to fill RAM with dirty disk-backed pages that only later get written out to disk. When your system is in this state, accessing even activity used pages (say, your window manager's heap) might incur be a slow hard fault.

(OSes have gotten a little more resilient against this scenario over the years, but it illustrates the issue.)

Memory pool balancing is a really hard control theory problem! I don't blame some people taking the RAM size efficiency hit and just turning off swap entirely. I just think it's a shame to have to resort to that extreme.

[vardump]

I wonder if ARC can be used for replacement policy:

https://en.wikipedia.org/wiki/Adaptive_replacement_cache

Although I guess it's patent encumbered...