[tawy12345]

The rationale for disabling overcommit only really makes sense if physical and virtual memory consumption are in the same ballpark. That's true for some workloads but not generally true.

There are totally legitimate use cases for processes that use significantly more virtual memory than physical memory (since virtual memory is relatively cheap, but physical memory isn't). A lot of programs are going to touch and not release all virtual memory they allocate from the kernel, but there are plenty of important counterexamples.

fork()/exec() is one example (which I've been burned by personally), but there are plenty of others. Any program that uses TCMalloc and has fluctuating memory consumption will have a lot of virtual address space allocated but not backed by physical pages. Sophisticated programs like in-memory caches or databases can also safely exploit a larger virtual memory space while keeping the amount of physical memory bounded.

[bjourne]

Virtual memory and overcommit isn't the same thing. Virtual memory means using disk space as memory. Overcommit means that the OS allows more memory to be allocated than it can guarantee is available. It is exactly like an airline booking 1 000 passengers for a flight with 500 seats, hoping that only half of them will actually show up. I don't think overcommit is ever needed in a modern system or is even a useful feature. For example, Windows doesn't support it at all.

[thayne]

> I don't think overcommit is ever needed in a modern system or is even a useful feature.

If you want to spawn child processes from a process that uses half the system's memory (not uncommon in server environments) using fork/exec it is useful. In case you aren't familiar with how that works, the parent process makes copy of itself, including a virtual copy of all the memory assigned to that process. That memory isn't actually allocated or copied until the child process child tries to write to it (and then only the specific pages that are written to). Typically, the child process then calls `exec` to replace itself with a new program and replaces the process memory. Without overcommit or swap if the parent process is large enough, then the fork syscall fails due to insufficient memory.

In a desktop environment using swap/virtual memory is fine. But in a server environment, where the disk may be network-attached (higher latency) and just big enough for the OS and applications, needing significant swap space is often undesirable.

[wahern]

Windows supports overcommit, it's just not the default, and it's not how typical Windows runtimes allocate memory. And the nice thing about making it opt-in is that processes which didn't ask for overcommit won't get shot down when overcommitted memory can't be faulted in.

[bjourne]

No, it doesn't. See quotemstr's explanation here https://lwn.net/Articles/627632/

[wahern]

Thank you for the clarification.

What's left out, though (and perhaps the source of my confusion), is that you typically commit a reserved page from an exception handler, and if the commit fails then presumably in the vast majority of situations the process will simply exit. See https://docs.microsoft.com/en-us/windows/win32/memory/reserv...

If the code dereferencing the reserved-but-uncommitted memory was prepared to handle commit failure beforehand it would normally have done so explicitly inline. I can't imagine very many situations where I would pass a pointer to some library expecting the library to handle an exception when dereferencing it. There are some situations--and they're one reason why SEH is better than Unix-style signals--but extremely niche.[1]

Either way, my point was that Windows does strict accounting, and while you can accomplish something like overcommit explicitly, nobody else has to pay the price for such a memory management strategy. Only the processes gambling on lazy commit end up playing Russian Roulette.

[1] In a POSIX-based library I once used guard pages, SIGSEGV handler, per-thread signal stacks, and longjmp to implement an efficient contiguous stack structure. This was in an extremely performance critical loop (an NFA generated by Ragel) where constantly checking memory bounds on each push operation had substantial costs (as in multiples slower). AFAICT, it was all C and POSIX compliant. (Perhaps with the exception of whether SIGBUS or SIGSEGV was delivered.) Though, because neither POSIX nor Linux support per-thread signal handlers you could effectively only use this trick in one component of a process--you had to hog SIGSEGV handling--without coordination of signal handlers. SEH would have resolved this dilemma. This being such a niche use case, that wasn't much of a problem, though.