[andreasvc]

I still don't comprehend why one needs swap at all. All the explanations I have come across talk about not having enough memory. Given that one has at least 8 GB of memory, or maybe even >100GB, why on earth would you need swap? Sure some process might allocate even more than that, but maybe it's better to refuse such a request than to slow down the whole system due to thrashing.

I get the idea that the reason might be that a lot of programs allocate memory which they don't actually need regularly, which is then very convenient to swap out. Rather than enabling this bad habit using slow disk storage it would be much better to expect programs to be more frugal, or at least signify whether something should be kept in memory or not.

[lobster_johnson]

The problem is that if you disable swap completely and let the system refuse allocation, you will face failures not just from the process that was responsible for taking most of the memory, but from existing ones that attempt to continue operating normally -- including innocuous processes such as bash and sshd.

This would affect C programs in particular, since they usually manage their memory manually. If bash can't malloc() a buffer for its input, for example, it will simply fail, and you might be able to do anything to fix the system; the same goes for sshd, which might end up refusing new connections as a result. Programs that preallocate important data structures, and programs using garbage collection, would fare somewhat better.

In other words, if swap is disabled you will still need a sort of soft limit or reserved space to ensure that programs can survive memory starvation. I don't know if the Linux kernel (or the GNU C library) has anything of the sort.

[jeremycole]

Well, for the systems in question, they don't necessarily need swap, and they may never actually swap in practice aside from problems such as the NUMA issues described. However, even with swap disabled, the NUMA issues cause problems with performance due to running a single node out of memory.

[makmanalp]

Because most data is critical and you can't afford to just drop it on the ground whenever you please. A better option would be to have the application/db to have its own swap routines optimized for its own purposes rather than letting the OS doing a catch-all swap method.

[jcrites]

Even beyond that: it's desirable for a machine to be able to compute arbitrarily large data sets. If the data set can't efficiently fit in memory, the machine should still make progress, just more slowly, using disk.

It is not desirable for a machine to have a "wall" which, upon being hit, becomes a harsh restriction on its capabilities. This is because we often encounter the "wall" unexpectedly, at a time that might be critical.

[vidarh]

But they still have a wall - it's just takes a bit more to hit it. In Linux systems swap is usually 2x memory. With swap set like that, all swap does is raise the wall to 3 times what it previously was.

But for a lot of systems your service will fail shortly after you start swapping anyway, because the performance cost of swapping is so high that it often starts a death spiral (can't handle enough requests, so they start piling up, eating even more memory, until your system dies or you hit connection limits etc.).

So "best case" in a typical configuration is that the wall is a bit higher. Worst case you gain nothing at all from the swap.

Personally I treat it as a failure if we ever hit swap - it means connection limits etc. has been set too high.

[GoodIntentions]

"Personally I treat it as a failure if we ever hit swap"

/agree. but still a useful feature.

The degraded performance a system will show when it starts hitting disk instead of memory is a great 'soft' failure.

I think it is good to have graduations. Going from 'OK' to 'Damn-this-is-slow' before 'Fail' is handy.

[ars]

That's not why you need swap. Swap is because many applications will use memory when starting and then never touch it again.

You can therefor swap it out and use the extra memory for cache.

Most long term applications only need a small fraction of their startup memory.

[justincormack]

Then surely they could free it?

[ars]

If you allocate memory "after" that memory, it's not possible to return the earlier memory to the OS.

Also, suppose you need the memory only for startup and shutdown (things like logfiles, network connections, command line parsing, etc).

[__david__]

I like the erlang technique of just dying when something bad happens. That is, if you write your app right then you can afford to just die at any point (without dropping any data on the floor at all). I agree with the grandparent--I've started to disable swap on my production servers because I reason that if I run out of ram then I haven't I configured something correctly. A real server shouldn't ever swap--heavy swapping grinds the whole world to a halt which means requests are being serviced way too slowly, if at all...

[jws]

An application that is aware of the half dozen of so caching layers from register to platter can perform dramatically better than a naive program. Two wrinkles:

1) it needs to either be told the various sizes, speeds, and quirks on each server to make best use. (just some work)

2) it needs to coordinate with the other processes running on the system to divide up the resources. This is hard. Generally people bail and just assign some share of RAM and hope for the best with the other layers.

[andreasvc]

I agree of course, but I think the current situation is worse. Applications can allocate more than the amount of physical memory, at which point the system can become unusable. If on the other hand an allocation would have been refused at an earlier stage, there would have been no critical data to drop.

I guess I want to argue that with the currently typical amounts of RAM, all critical data should fit in RAM and stay there. The idea of virtual memory was to abstract over the difference between RAM and disk, but perhaps this has become a harmful abstraction now that RAM is big enough while the disadvantage of slow disks remains. RAM and disks are fundamentally different parts of the memory hierarchy, and should be treated completely differently by applications.

[jeremycole]

There's also copy-on-write to consider. If you actually had to allocate real memory-backed pages for every allocation a process made, had the right to, but never modified, the size of each process would be a lot larger. For example, if a process consumes 1GB of memory and forks a child, the child has access to the same 1GB of memory, but it doesn't really consume 1GB of memory. It has mapped a bunch of copy-on-write pages from its parent, and when either of them next modify those pages, the real memory is allocated.

If they never modify those CoW pages, they can both happily keep using the same copy of the page in memory, and you can have two 1GB processes using a total of e.g. 1.01GB of real memory.

This is very useful in practice, but it means that the system needs the ability to over-commit memory allocations (allow CoW allocations etc., when there is no actual memory available to back it), and over-commit currently, and probably should, requires swap (some place to dump pages in case an over-committed allocation comes calling).

[lawnchair_larry]

Overcommit does not use or require swap. You can have overcommit enabled on linux and not be using swap at all.

[wmf]

But then you may meet the OOM killer.

[lawnchair_larry]

Again, that has nothing to do with swap being enabled.

[bifrost]

You don't need it on some systems, however Linux will typically kernel panic if it has no swap rather than just killing processes. FreeBSD/NetBSD/Solaris all do this properly, however its still good to have swap because having your processes die when you make a tiny mistake is rather irritating. This is a great example of why you need experienced ops people to reign in the ideas of green engineers.

[toddh]

Swapping made sense when we had computers with limited resources running multi-user workloads. Neither is now true. These servers are special purposed so they don't need the overhead of a general OS at all. But aren't there two issues here? Swap and node allocation? Even without swapping wouldn't you have NUMA node allocation issues?

[andreasvc]

This is what I suspect, but in practice it seems not a good idea to just turn off swap:

> DO NOT TURN OFF SWAP to prevent this. Your box will crawl, kswapd will chew up a lot of the processor, Linux needs swap enabled, lets just hope its not used.

(from one of the blogs linked in the article).

However, I can't find a clear explanation of why this is so.

[lawnchair_larry]

That was an old bug. It is perfectly fine to disable swap completely. It's the first thing we do on our servers, partly due to the issues detailed in this blog post, and partly due to it being archaic and useless when you have 128GB-512GB of ram.

[jeremycole]

Yes, they are two distinct issues which are somewhat conflated: the actual "swap insanity" and swap in general, and NUMA performance effects. I think for my purposes, the negative effects of NUMA are relatively hidden, so we only see the side effects. That is, it's impossible to know at this point whether optimization for NUMA would help MySQL, since no one has made a serious effort to do it.

I suspect it would help quite a bit, if done right, and for the right query workload.

[ataggart]

The two may be less distinct in the face of some approaches to minimizing remote reads/writes, e.g., replication of pages across domains. Apart from ensuring the cost of such replication is warranted, the dominant pressure against replication is the effect it has in increasing the memory footprint of the application, and thus possibly triggering more page faults.