| Do you have a swap partition? I've been in a couple of interesting discussions about Linux memory management lately that enlightened me somewhat, and I won't claim to be an expert now, but I've been around the low-memory block enough to understand now that, there's no simple right answer to the question of "Do you have swap?" "The Linux kernel has overcommit baked into the fiber of its being." I've begun to understand that this idea is so deeply engrained in the kernel that in a multi-tenant or desktop workstation, you simply can't extract it back out and "just provide enough RAM," unless you know the performance characteristics and you really mean it when you say "that should be enough RAM." If you don't have any swap and the kernel starts to run out of memory, it's going to start evicting whatever pages it can back to disk. (Wait, pages back to disk? I told you I didn't have swap) Yes – the linux kernel can page things back to disk even if you don't have swap, remember all of the binaries you're running have originally come from that disk, and the kernel knows it doesn't strictly need to have them in memory until they are volatile, or you tried to read those pages again. Having some swap gives the kernel something else to evict, so have a healthy amount of swap and Linux will find the occasion to use it for the least frequently used pages that are not already on disk. This will improve your "nearly out of memory" performance. The second worst thing that you can do is put your swap on fast SSD or NVMe, and it's not why you think. The kernel is making decisions based on a heuristic which is complicated and well-documented, but inscrutable. If the solid disk is 50x faster than the spinning disk that the swap was originally designed to use, then swapping will cost less overall and the heuristic will lean on it as a strategy to keep the OOM killer away even more often. You may find your cache recycle rates going through the roof because things can be paged out to disk and re-loaded faster than should be possible. I don't fully understand this part, but I suspect the answer is "try to use Swap less, and be aware of when you are using it." The kernel does really not want to kill off your processes, and it has more opportunities than ever to ensure it keeps too many balls in the air when you have asked it to do so. So, find a way to stay ahead of the kernel and know better. If you have a dock widget that tells when you are going above 50% swap usage, you can close some tabs before it gets to be an unrecoverable situation. It's a mystery to me why modern computers don't come with disk activity lights, as this problem we didn't need dock widgets to solve 20 years ago when literally every computer came equipped with one. The best advice is to have enough RAM for whatever you're doing, and at 32GB "I think you've had enough." At any rate the one suggestion that I could give is, if you anticipate running out of memory (ever, and it looks like you still do), then you should be sure to have a healthy amount of swap, to me that's probably at least 5 or 6GB but YMMV. But, 32GB for a desktop workstation really ought to be enough IMHO, so try to find a way that you don't run out? If you're eating all that memory up with VMs, try a lighter weight solution for your ephemeral workloads like footloose, which behaves like a VM in the ways you generally tend to want for your dev workloads, (like for example, it can run systemd like your deploy target most likely does, if you're using VMs to match the deploy target). Footloose doesn't impose the "VM's" whole footprint upfront due to actually being a container, so when you run out of memory it will be because your application workloads used too much, not because your virtual machine manager has grabbed much more than it needed. |