[lnkuiper]

I'm not sure if I understand, but we did not use a memory map (mmap), but rather blocks of memory that are explicitly (un)loaded by the buffer manager.

The M1 + SSD performs really well here. We tried to an external sort experiment on the x86 machine, but the SSD is old and only has a write speed of 150MB/s (compared to the MacBook's 3GB/s) and it was incredibly slow. So you definitely need a fast SSD for this.

The columns we chose to sort by are rather arbitrary, but we shuffled the table before running the experiments to make sure there is no ordering left from the generation in there.

I like the merge sort trick you described!

[nextaccountic]

> I'm not sure if I understand, but we did not use a memory map (mmap), but rather blocks of memory that are explicitly (un)loaded by the buffer manager.

But then, do you have your own cache competing with the OS cache? Or you use O_DIRECT to disable the filesystem cache (at least on Linux)?

[tomnipotent]

Most databases rely on OS filesystem cache, which is very compatible with the way buffer caches manage data with fixed-size pages/blocks.

[jandrewrogers]

Some databases rely on the OS file system cache, this is true. However, it is not controversial that doing so reduces performance by integer factors. Bypassing the OS file system cache is idiomatic in database kernel designs where absolute performance is a key objective.

[tomnipotent]

Using O_DIRECT and other features still makes you reliant on the OS cache, even if all you're doing is keeping it empty and out of your business. I've only run into a few that manually manage block devices directly.