[gumby]

Sure, you push the failure into the processes, and most OSes I've used do this. Thus (to use unix terminology) sbrk and fork() fail and it's the program's responsibility to handle that, gracefully or not as it wishes. You also degrade more slowly via paging.

You shouldn't get to the point where a perfectly innocent process is killed to reclaim memory. A process can die, or even (in some cases) recover cleanly from running out of memory -- that should be the process's choice.

[throwawaylinux]

Which OSes have you used that do that?

In Linux you can do that -- disable overcommit (although I'm not sure if that's 100% foolproof). But that's often not preferred because that way does lead to "perfectly innocent" processes being killed: whoever allocates the next page when memory has run out loses. And how do they get to decide? What if a failure path requires paging in one more page of code from disk? Or causes a store to a COW page?