[hmry]

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class.

I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author also shared some unconventional opinions about undefined behavior (like "Compilers are deterministic, if I know what platform I'm compiling to then no behavior is undefined. And if compiler authors disagree, they are morons.")

But presumably it's thoroughly tested, so those aren't problems in practice? Would be really interested to hear from people who've actually used it. I've mainly stuck to SQLite instead.

[markasoftware]

Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb.

And yes, I ensured there were no outstanding long lived readers, verified with mdb_stat -r. My workload used one transaction per read/write anyway (never needed larger atomicity). Once the db got into the bad state, running my program on it would almost immediately run into the issue again, so I really think the db is in a bad state such that most writes would cause it to hang, not related to how I do transactions. This workload would pretty consistently hit the issue once the db got to several hundred gb.

Issue #10236 on the OpenLDAP bug tracker might be the root cause, who knows. It's been marked CONFIRMED for years without a fix, while other similar issues are created.

This is extremely annoying. It seems workload dependent (other workloads I've run create absolutely massive lmdb dbs without this issue) and once it happens your only recourse is to make a new db and copy the contents over (thankfully reads still work fine on these borked dbs).

Other than that, though, it's great. Never in any case had actual data corruption, and reads and writes are extremely fast (until this issue happens)

Edit: fun fact, since shopify may have created Bolt in response to this bug, and then Bolt was the root cause of the 73-hour Roblox downtime in 2021, this bug may indirectly have caused one of the worst outages ever!

[jnwatson]

I've used LMDB in production for multi-terabyte databases, and we encountered the long-write time but found a solution.

The important idea is that LMDB offloads cache management almost completely to the OS. You have to become intimately familiar with the way that the page cache works and how to configure it.

[uroni]

That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.

Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there. I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.

Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.

[hyc_symas]

By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.

[markasoftware]

FWIW I had this issue even with the MDB_NOSYNC flag so it shouldnt be force flushing to disk unless I'm out of ram or whatever

[hyc_symas]

LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.

[keynha]

Dropping the explicit P_DIRTY flag in 1.0 is a neat change. What tracks which pages still need to be written back at commit now that the flag is gone?

[hyc_symas]

The txnID was added to the page header to enable support for incremental backup. As a consequence, it's sufficient to compare a page's txnID to the current txnID to know if it's dirty or not, and spilled pages don't need a cleanup pass to clear their dirty bit on commit so commits of large txns are faster now.

[ChrisTrenkamp]

I can't go into specifics, but I use LMDB for the commandline application I maintain for my employer. I also extended it into a web service for internal use. As long as you stick to the safe LMDB options, which are the default options, it's reliable. The documentation clearly outlines what safety guarantees you lose when you enable/disable certain options: http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5...

I had a situation where the web service's writes were slowing down to an unbearable crawl because the number of entries in the database were reaching tens of billions of entries. Thankfully, the users never experienced the slowness. The website stayed nice and fast, even though the background updates were extraordinarily slow. The issue was fixed by sharding the databases.

[alfiedotwtf]

> And if compiler authors disagree, they are morons

I remember arguing with Howard years ago on “C vs Rust”. He said that you don’t need Rust, you just have to be good at C programming, so I pointed out CVEs in LMDB attributed to his own bare hands… so there’s that.

[Kerollmops]

I recently talked to Howard [1] about lies he was saying about Sanakirja, an LMDB-inspired disk allocator. That's always the same arguments: C is better than Rust for X, Y or Z reasons. While I reported a segfault just two weeks earlier... [2].

I love LMDB, we use it in Meilisearch (second most stared search engine on GitHub) [3] for about 7 years now. The main issues were related to write speed but we do a compaction of the database and write performances are way better after that. We never had any major DB corruption... I mean... other than when using it on Azure. Azure never works, that's expected, I suppose.

[1]: https://mastodon.social/@hyc/116838499082046918 [2]: https://bugs.openldap.org/show_bug.cgi?id=10522 [3]: https://github.com/meilisearch/meilisearch

[alfiedotwtf]

Unfortunately, yeah.

> The main issues were related to write speed but we do a compaction of the database and write performances are way better after that

I'll ping you if I ever get around to rewriting a faster kv-store in Rust :)

[Kerollmops]

Actually, you should take a look at [1]. It's made in Rust, inspired by LMDB, and supports a cool feature that allows close to anything to be implemented: allocating any page you want to store anything you want. The BTree storage is optional and you can implement whatever storage system you want. When storing a value to disk, you can allocate pages and decide exactly how you plan to store the bytes, allowing you not to store the length of them or to split your data into multiple pages, etc.

[1]: https://pijul.org/posts/2021-02-06-rethinking-sanakirja

[erikschoster]

I use it as a session store for a computer music system. It has worked well for me as a way to read mutable (by any client) parameters during synthesis, clients will often read dozens of parameters during a block of computation (a relatively short window of time in the low milliseconds typically) without adding any noticeable overhead to the render time for each block.

Edit: I also tried using it for larger blobs of data (like audio) but ended up only storing a reference to shared memory for larger blocks, anything larger than IIRC 4k that can't be stored in a single node kills performance, but for small values it seems pretty great.

[radiator]

It has been used successfully as the backend for OpenLDAP and Monero, at least.

[thombles]

Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.

[hyc_symas]

Irrelevant, since LMDB doesn't dirty the pages in the mmap. The mmap is read-only.

[senderista]

Isn’t that why the mapping is read-only by default?

[zbentley]

Wow, really?

Then what’s the point of memory mapping in the fist place? Or do they suggest manual flush/sync actions for persistence.

[tynorf]

IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync.

However, some people think there are problems with this usage: (pdf warning) https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf

[senderista]

How is pwrite/fsync any better than mmap/msync? Both go through the page cache and combine asynchronous writeback with forced flush. One theoretical advantage of pwrite might be that you can handle I/O errors, but I’d like to see a case where recovering from an I/O error makes sense (rather than just crashing the database, which SIGBUS would do anyway by default).

[hyc_symas]

write/fsync can be faster in a large dataset because writes let the filesystem know an explicit list of dirty pages, so fsync only needs to deal with them.

mmap/msync gives no hints about which pages are dirty (unless the app tracks them itself and msyncs them individually, which would completely defeat any reduced syscall advantage of using a writable mmap in the first place) so the entire map must be scanned for dirty pages.

In practice, the expected performance advantages of using a writable mmap just aren't there, and coupled with the ease of silent corruption, it's best to never use that approach.

[emmelaich]

There's a project to have LMDB as a backend for sqlite. It originates in the LMDB author's (Howard Chu) own work.

https://lumosql.org/src/lumosql/doc/trunk/README.md

Also https://www.actordb.com/ uses LMDB with a sql queries. (Not sure if sqlite or not)

[jimis]

If you use the default mode MDB_SYNC then it's reliable but can be slow for writes. For max write performance you need MDB_NOSYNC (IIRC that's what the official benchmarks use) but then the whole database can be unrecoverable in case of crash. It has happened to me.

Sqlite in WAL mode will never lose all your data and performance can be configured vs durability by setting pragma synchronous to full or normal.

[nomel]

I think lmdb is mostly unusable, for many use cases. I switched to libmdbx, which fixes all the issues [2] I (and most sibling comments) ran into with lmdb.

[1] https://github.com/Mithril-mine/libmdbx#improvements-beyond-...

[OrangeDelonge]

We evaluated it but chose RocksDB instead

[packetlost]

I believe at least one of the two official Minecraft implementations use it for their map/save format.

[bch]

> stuck to SQLite instead

Different level of abstraction. I don’t think it’s highlighted enough either - this latest (1.0) and the previous 0.9.x are mutually incompatible, requiring essentially a dump/restore. It is mentioned (I forget which file ottofmh), but should be a

*HIGHLIGHT* in the CHANGES.

[ozgrakkurt]

It is a small amount of code so easy to integrate into an application.

It is really reliable except write performance in my experience.

Author of it writes very spicy stuff and sounds pretty rude.

I would recommend doing a prototype with real data scale and testing if it meets your requirements. The write performance can be really atrocious and It doesn't have a high performance potential because it is based on memmap.