> Our new engine is based on intercepting SQLite's disk writes and doing clever stuff with them. It was so easy because the file format is not just well designed but amazingly well-documented.
Quite the hobby project for a lot of people too! Other folks doing this:
I'm sure there's a lot of really cool local-first databases out there, but SQLite has the benefit of being incredibly widely battle-tested, with literally billions of installations worldwide. It has received thorough security research and fuzzing (it's part of Chrome's attack surface after all). And there's tons of resources online to help people understand how to use it. Although I'm sure there are alternatives that serve certain use cases better it's hard to imagine anything coming close for ours.
That said, the storage engine we've built is not that heavily dependent on SQLite specifically. Any database that uses a write-ahead log like SQLite does should be possible to adapt to it in the future. So maybe we'll eventually open it up to a variety of choices, or even let you bring your own as a Wasm module.
Oh, I've been informed that DuckDB uses SQLite under the hood, so maybe compiling DuckDB to Wasm and running it on top of this will be possible, we'll see.
> DuckDB is indeed a free columnar database system, but it is not entirely built on top of SQLite. It exposes the same front-end and uses components of SQLite (the shell and testing infrastructure), but the execution engine/storage code is new.
Correct. DuckDB is really interesting technology, but it's not a direct successor to SQLite for transactional workloads. It's also very new: there's a LOT of new code in DuckDB on top of the (heavily fuzzed) SQLite parts.
(I use it personally, but it's not the same thing as what we're building with D1)
Also, for OLAP workloads there's Workers Analytics Engine. Analytics Engine is arbitrarily scalable as opposed to a D1-like solution - I can almost guarantee I've inserted more data into its internal variant (which is built on extremely similar underlying infrastructure) than just about any customer would think of doing, and it handles it like a champ.
Basically, we already have a product for doing DuckDB-like things, and the D1 architecture isn't great for high volume OLAP workloads.
Disclaimer: I'm an engineer at Cloudflare, but not on a developer platform team. I'm not speaking for our developer platform strategy or anything, I'm just commenting on how it looks from where I sit.
Quite the hobby project for a lot of people too! Other folks doing this:
Rqlite https://hn.algolia.com/?query=Rqlite&sort=byDate , Dqlite https://hn.algolia.com/?query=dqlite&sort=byDate , Litestream https://hn.algolia.com/?query=Litestream&sort=byDate / LiteFS https://hn.algolia.com/?query=LiteFS&sort=byDate, marmot, mvsqlite https://hn.algolia.com/?query=mvsqlite&sort=byDate