| This is great to see, and a project I considered attempting myself for a bit. I’m excited to test it out. @jlongster I have a question about this: > The backend calls it [Atomics.wait] to wait on the result from the worker and blocks until it’s done. Does this mean the main (UI) thread is blocked during queries? Or are there more threads, like UI <- async messages -> SQLite main <- Atomics blocking -> SQLite FS backend? ———— At Notion, we’ve used IndexedDB for two purposes: (1) to durably persist a queue of changes to send to our backend, and (2) in the desktop app, to LRU cache the page data we read from the server to accelerate reads. Both of these used localStorage years ago, but we ported to IndexedDB because of data loss on localStorage. Porting was fine for the write queue, but we really noticed the slow when we tried porting the data cache. To get close to the original performance we coalesce reads, and we delay writes to the cache significantly so they can batch more effectively into a single readwrite transaction that we send after the reads for the current page load are complete. That worked okay, but it was annoying to maintain the IDB cache code because our Android and iOS apps used SQLite for their caches, and it’s so much easier to add new queries using SQL compared to writing IDB iterations - and it’s faster. So we switched to using native SQLite via a bridge to a Node process. Now with absurd-sql, maybe we could bring the same caching logic to browsers. The thing stopping me is how unreliable we’ve found IndexedDB to be - aside from the optimization work. We notice a lot of bugs in IDB implementations on different browsers. In Safari (especially on iOS) there’s a bunch of spooky issues that have caused stalls or spurious errors, sometimes requiring an app restart before the IDB database can be re-opened. Forget it on Android - weird vendor webview patches mean your storage might get cleared out from under you. On Firefox, we notice that sometimes the IndexedDB database doesn’t create all the object stores we request for some reason. Even on Chrome, IndexedDB can suddenly start refusing writes in the middle of a session with no clear explanation, and on Windows restarting the computer is sometimes the only fix. If we can share SQLite queries with our native apps then maybe it’s worth wading deeper into these issues… but it really does feel like building on quicksand. |
The latter! Your app running queries must be on a worker, and then the IDB backend will spawn another worker. `Atomics.wait` is not even available on the main thread.
Ideally in the future, there will be a better storage API that we don't even need all the Atomics silly-ness (hopefully it provides Sync methods)
That's really cool re: Notion! That's exactly the kind of thing I want too: a way to just build apps the same way everywhere, on mobile/desktop/web.
You are right about various issues, and I personally don't have to worry much about it on my app because I have native mobile apps and I don't support the web version on mobile. I intentionally do that -- the mobile web is just too broken in too many ways. My impression is the IDB is more stable on desktop, but because mobile is more memory sensitive there are more issues there.
However, you should try it out! I definitely discovered a lot of weird things; I definitely was able to get Safari into a weird state the required a complete app restart. Here's the thing though: I found ways around them. If you do a lot of read requests in a certain way, Safari will lock up permanently. However, if you make sure to wait until the `readonly` transaction is finished before starting a new one, the problem goes away. I was able to reliably reproduce that problem and it went away with that fix.
I think absurd-sql is so promising because it normalizes the patterns of how IDB is accessed, and it already includes fixes for a bunch of edge cases. There are probably more, but try it out! If you run into an edge case, we can tweak the IDB backend until it works. We can paper over these issues in the underlying backend and you don't have to worry about it because you aren't directly managing IDB read/writes.