[h4kor]

The main reason I use postgres instead of SQLite is that I have multiple processes accessing the database, often 1 web service for API/Website and a worker running in the background doing heavy tasks (e.g. image processing). Both need access to the database and SQLite will run into locking issues.

How do you overcome this with SQLite and Django?

[mrklol]

Afaik the fix for that is to have multiple read only connections and one write only connection.

[Sammi]

Yes by enabling the write ahead log feature: https://sqlite.org/wal.html

It's on by default in many sqlite drivers because it really is the best default. But it isn't on by default in upstream sqlite even though it's been out for ages now.

[sgt]

Sure but if you're dealing with WAL logs, why not just go Postgres? Then you also get a port you can connect to from remote machines if you need.

[wild_egg]

> "dealing with WAL"

What's there to deal with? You turn it on with a pragma and forget about it.

[sgt]

Sure but once you have WAL logs, you suddenly have a more heavy weight setup. Backing it up you'll want to back up those WAL logs to achieve proper point in time recovery, and so on. My point is, you're now bolting on extra stuff on it to do things that Postgres can do (which can be pretty light weight). Not disrespecting SQLite, still one of my favorite DB's.

[wild_egg]

What? Why are you backing up the WAL?

    sqlite3 source_database.db ".backup backup_database.db"

Now the WAL content is rolled into your new backup file. Stick a timestamp in the backup file name and run this as a cron job every N minutes and you have all the recovery you need. Another one-liner to sync to S3 and you're all set.

Edit: And just to clarify, that command can be run on a live DB as it's being used by your app server. SQLite handles external concurrent readers just fine.