Note to anyone who thinks they have "disabled swap": most programs, unless they go out of their way to remap themselves, are backed by a file on disk, and Linux will cheerfully discard the executable text pages of the program to reclaim, and then immediately have to read them back in from disk. This is at least as bad as swap, and worse in some ways.

One of the key observations is that you should basically always run your applications with some memory limits so that the system and management services will remain responsive etc even if the application is misbehaving. Luckily stuff like docker and systemd makes setting limits pretty easy

TLDR:

- OOM will swap or trigger the OOM killer but Linux might try to reclaim memory from caches before and slow down (using a cgroup limit solved this).

- glibc malloc can easily cause memory fragmentation with many threads (switching to Jemalloc solved this)