Hacker News new | ask | show | jobs
by gchamonlive 26 days ago
Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?
4 comments

Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.
I swear it didn't occur to me that that mean WAL, makes much more sense now LOL
I assume they meant a log like a WAL. A WAL should be (quite literally?) all you need for durable workflows.

A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.

Is it common to use logs as a proxy for write-ahead logs?
Folks this is meant to be an honest question, not a snarky comment. I'm not a DBA, I'm DevOps/SRE and logs for me always meant execution logs. I'm just curious if between those involved in database domain logs is used to refer to WAL.
I think the original poster in this thread was joking. A fair number of databases use "logs" as a core mechanism for storing and sorting data. "Logs" in this context is not to be confused with stdout/stderr output that you may collect from a running program and forward somewhere like Cloudwatch/Elasticsearch etc. "Logs" in the context of databases here refers to the data structure; which can generally be defined as simply an append-only "file" (I put file in quotes because just because something is a log does not mean it is necessarily written to disk yet - that's why write-ahead logs exist). It's not just write-ahead logs.

Google "Log-structured merge trees" if you want an interesting read.

I read the parents comment as sarcasm and not a serious suggestion.
Log as in the structure.