Hacker News new | ask | show | jobs
by bob1029 1405 days ago
I am curious what the situation is where the recovery log grows to large size and what the actual consequence of this would be.

We have been using SQLite in WAL mode for over half a decade and never witnessed this. Several of our databases can see concurrent access from hundreds of users with transactions in the 1-10 megabyte range, so I find it a bit odd this never came up.

4 comments

> I am curious what the situation is where the recovery log grows to large size

From the link, "if a writer writes to the database while a checkpoint is ongoing [...] it does mean that the wal file may grow indefinitely if the checkpointer never gets a chance to finish without a writer appending to the wal file. There are also circumstances in which long-running readers may prevent a checkpointer from checkpointing the entire wal file - also causing the wal file to grow indefinitely in a busy system."

> what the actual consequence of this would be.

SQLite runs in a surprising number of places. In an embedded environment disk space may be limited.

I've seen the WAL grow when there's a constant write load and a long background read transaction keeps an old version alive. If you also never reach a point where there are 0 connections to the DB, you can keep this large WAL file around (most of it is unused) for a long time.

I've had to restart services when WAL files had grown to multiple GBs and wouldn't shrink.

I ran into this when I was importing about 1TB into a DB while simultaneously reading from it and performing tasks. This was all in a dev environment, but it did come up.