I’d say, let the one who tried to allocate memory crash, and if you’re a critical process like xlock, use statically allocated memory and don’t alloc again.
This is only a viable answer when overcommit is disabled. The problem comes when overcommit is enabled and you find yourself in a position where many programs think they already have memory and yet there is none to give them. If you simply kill the first piece of code that encounters the end of available memory you might take down anything including the kernel itself.
Nothing like statically allocating memory can work when overcommit is enabled because the kernel is free to compress memory, page it out and etc. and then murder you the next time you try to perform any operation that it doesn't have the space for, no matter how safe and static your initialization was.
Note that overcommit is very useful in many cases including the ones where swap saves the stability of the system under conditions that would otherwise completely lock up or panic, so it's also not viable to just prevent it from being used.
OOM killer always felt like a band-aid on a severed artery to me. I've rarely seen a machine that got into OOM state really recover without a full reboot.
Why would a system break if you SIGKILL a process?
I’ve seen plenty of server log with OOM killing mariadb processes, and then being restarted automatically by systemd, often with no one noticing if not days later.
The thing that bogs down systems and often makes them unrecoverable is when a memory hungry process starts swapping. Good luck trying to SSH in. Swap is such a silly idea on servers - good to deal with pages no one accesses, catastrophic when you’re out of RAM and memory latencies suddenly become 4 or 5 orders of magnitude slower.
IMO if the security of a system depends on the lock screen not crashing then the system is not very secure. Security protocols should never fail open like that; a lock screen should never simply be a layer on top of the authenticated desktop. Windows and macOS get this right. I believe Wayland display managers are also able to get this right (but I haven't checked).
I don't know why X11 didn't just add an extension that a client can enable saying "if this client exits unexpectedly/uncleanly [without disabling the extension], just kill the X11 session".
Wayland the protocol already fixes this -- there's nothing that exactly requires a display manager to not have a completely separate desktop for the unauthenticated state, where a trusted application (or the display manager itself) can accept credentials in order to authorize a transition to the authenticated state, and where a crash of the trusted application or lock screen does not result in access to the authenticated state. I just dunno if anyone does that yet. I'm sure somebody must have...
> Granted, then you have a locked screen that the user may or may not be able to unlock, which is awkward if better.
The most secure system is one that cannot be accessed, technically. In some cases it's better not to let anybody in than to let an attacker in (technically). Of course, this is frustrating for the user.
Statically allocated memory can still OOM on access, due to overcommit and lazy page table population. What you really want is mlockall(2) (probably with MCL_CURRENT|MCL_ONFAULT followed by madvise with MADV_POPULATE_*)
oops MCL_ONFAULT kinda does the opposite of what I wanted - I think if you omit that you can skip the madvise, and mlockall will populate everything for you.
The problem is that Linux has memory overcommit and it will OOM when a process faults a page in, not just when someone allocates memory.
So the OOM condition can hit any random process, not necessarily one that just tried to allocate. If you don't have some sort of selection, then you would still have an OOM killer, only it will be killing completely at random.
Nothing like statically allocating memory can work when overcommit is enabled because the kernel is free to compress memory, page it out and etc. and then murder you the next time you try to perform any operation that it doesn't have the space for, no matter how safe and static your initialization was.
Note that overcommit is very useful in many cases including the ones where swap saves the stability of the system under conditions that would otherwise completely lock up or panic, so it's also not viable to just prevent it from being used.