[hobo_mark]

Tangentially related: I'd like to use litestream but my SQLite files are several gigabytes, is there a way to lazily download the db only once it's being accessed? (using something like userfaultfd maybe? just thinking out loud)

[pkhuong]

Verneuil (https://github.com/backtrace-labs/verneuil) offers that for read replicas, because backtrace as well has some multi-GB sqlite DBs. It's a VFS (loadable at runtime as a sqlite extension), instead of a filesystem.

I don't remember if that's in the docs; commit that adds the configuration field https://github.com/backtrace-labs/verneuil/commit/027318ba74... and commit for the enum https://github.com/backtrace-labs/verneuil/blob/e6697498f3ba...

(the actual implementation is nothing special https://github.com/backtrace-labs/verneuil/commit/b6bdfcf7bc...)

Issue for the feature: https://github.com/backtrace-labs/verneuil/issues/12

[benbjohnson]

I'm not sure how that could be implemented with Litestream since it runs as a separate process. It could be possible with LiteFS where it just pages in data on the fly. That's on the roadmap but it's probably still a ways out.

[mandeepj]

> is there a way to lazily download the db only once it's being accessed?

Not sure what's your full scenario, because you mentioned "lazily download" so thought you might have a luck here https://news.ycombinator.com/item?id=27016630

tl;dr: using http range and a smaller page size - might be the way to go