Hacker News new | ask | show | jobs
by thombles 30 days ago
Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.
3 comments

Irrelevant, since LMDB doesn't dirty the pages in the mmap. The mmap is read-only.
Isn’t that why the mapping is read-only by default?
Wow, really?

Then what’s the point of memory mapping in the fist place? Or do they suggest manual flush/sync actions for persistence.

IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync.

However, some people think there are problems with this usage: (pdf warning) https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf

How is pwrite/fsync any better than mmap/msync? Both go through the page cache and combine asynchronous writeback with forced flush. One theoretical advantage of pwrite might be that you can handle I/O errors, but I’d like to see a case where recovering from an I/O error makes sense (rather than just crashing the database, which SIGBUS would do anyway by default).
write/fsync can be faster in a large dataset because writes let the filesystem know an explicit list of dirty pages, so fsync only needs to deal with them.

mmap/msync gives no hints about which pages are dirty (unless the app tracks them itself and msyncs them individually, which would completely defeat any reduced syscall advantage of using a writable mmap in the first place) so the entire map must be scanned for dirty pages.

In practice, the expected performance advantages of using a writable mmap just aren't there, and coupled with the ease of silent corruption, it's best to never use that approach.