[simonz05]

The paper analyzes how file systems and PostgreSQL, LMDB, LevelDB, SQLite, and Redis react to fsync failures. It shows that although applications use many failure-handling strategies, none are sufficient: fsync failures can cause catastrophic outcomes such as data loss and corruption.

[XorNot]

This sounds a lot like we need to come up with the correct API for this and switch to it.

[GordonS]

I wonder if such an API would require hardware support in order to remain performant?

[eis]

I'd argue some modern hardware e.g. NVME SSDs already expose a hardware interface that would let a new transactional storage API be faster than what we have right now. Unfortunately it is a rare example in a see full of specs that do their best to avoid providing clear semantics and guarantees like atomic sector writes.

I guess the next study would need to look into what manufacturers actually fully adhere to the spec. They've not exactly shown best behaviour in that regard in the past (lying about fsync etc) :(

[xyzzy_plugh]

Without the hardware being open, how do you prove this? In a past life I concluded and argued that there's no point in assuming hardware tells the truth. You have to expect the worst and have a plan for handling it, even if that plan is giving up and failing.

Importantly this means you can't claim stronger semantics around e.g. atomicity. You can certainly work around most if not all issues, using redundancy, verification and distribution. But in isolation you cannot even properly observe extreme failure scenarios, you can only reduce their probability, and even that is limited.

[eis]

Fully agreed. You can never prevent all issues because there's always a chance for a fire, bitrot due to cosmic rays etc and thus you can only reduce the probability as you correctly concluded.

Unfortunately the world we are in right now just makes the hardware issue even worse with APIs that are prone to introducing bugs in the programs.

Bad hardware should result in the software being extra carefully crafted to balance it out a bit but somehow we ended up with bad hardware and bad software :(

[hyc_symas]

The paper's analysis of LMDB is wrong.

https://news.ycombinator.com/item?id=32462537