[hot_gril]

I've seen the same. It will sometimes prefer to swap rather than evict the disk cache.

I don't know how Linux does this in particular, but intuitively swapping can make sense if part of your allocated RAM isn't being accessed often and the disk is. The kernel isn't going to know for sure of course, and seems in my case it guessed wrong.

[kevincox]

At a very abstract level there isn't much difference between dropping a disk cache and writing anonymous memory to swap. Obviously the actual step of writing to swap is more expensive but once the data is out of memory they will both require a read to get the data back. So if you imagine that you are occasionally reading a file and an anonymous page you can thrash on either if your system doesn't have enough memory. After the initial write to swap it is effectively identical to keep reading either back in.

It is much better to swap out anonymous memory that isn't being used than to flush file data that is being used. Or another way of looking at it if you run out of memory you will thrash with or without swap enabled. The difference is that with swap the kernel can evict the most rarely accessed memory whether file-backed or anonymous. Without swap the kernel is forced to only consider file-backed memory, even if it is being used more frequently than the anonymous memory.