[derefr]

How do you feel about SQLite? Because when I read this architecture description, it mostly vibes with me, until I think about what happens to the data in the event of a power cut.

Is there logic in your app to potentially throw the last line away (incl. truncating it from the file) if it's invalid due to being the result of a non-atomic write? If so, seems a bit Not-Invented-Here compared to just using a (runtime-embedded) library that does that for you :)

[weaksauce]

> SQLite responds gracefully to memory allocation failures and disk I/O errors. Transactions are ACID even if interrupted by system crashes or power failures. All of this is verified by the automated tests using special test harnesses which simulate system failures.

from the sqlite about page. it's one of the most bullet proof and hardened pieces of software out there I think. it was made basically for exactly the ops use case of a file on disk. but who knows what the use case is for them. maybe writes to the db are few and far between so it's a fairly moot point.

[pavlov]

The AOF reader will discard and emit a warning about lines that can’t be parsed, but that’s the extent of it.

These apps are on Digital Ocean, and I don’t remember ever having unplanned downtime with them. They do sometimes migrate instances with advance notice, but that’s a clean shutdown.

I’m sure SQLite is a better choice for almost any app. My reason for not using it was to try avoiding dependencies out of curiosity, and also that I honestly really don’t like writing SQL — it just feels boring and error-prone. (Like eating celery, I know objectively it’s good for me.)

[derefr]

> My reason for not using it was to try avoiding dependencies out of curiosity, and also that I honestly really don’t like writing SQL — it just feels boring and error-prone.

Well, sure, but you can just read and write JSON blobs to a single-column table in SQLite. See also, "SQLite makes a better BLOB store than the filesystem does" (https://www.sqlite.org/fasterthanfs.html)

[eternityforest]

SQLite doesn't do "lines" like a text file, or truncate anything(Or, it probably does internally, but that's not how users think of it).

It's a real SQL DB with real records and transactions, and it is one of the most trusted and reliable pieces of software ever made. Like, check out the change log to get a sense of how they do stuff.

[derefr]

I think you misread what I said, or maybe didn't read the post I was replying to. I was pointing out that flat files have a problem with truncation; and that SQLite, despite being a very different thing than an append-only text file, can be effectively used as "an append only text file, but ACID." My question on how the GP poster feels about SQLite, is down to SQLite being the obvious solution to a problem I wasn't sure they realized they had — but also potentially still being "too many dependencies" for them.

[justinpombrio]

If you're not writing data very often, this isn't a concern. For example, if you assume 5 microseconds to write a line, and 1 line written per hour on average, then the chance that the power goes out while you're writing a line is 10^-9, i.e. will never happen.