Hacker News new | ask | show | jobs
by znnajdla 8 hours ago
Why are databases so hard?
1 comments

Because of the guarantees they provide.

Storing some data in a binary file isn't very hard. Making it so that you can do quick lookups on it (indexes) and implementing joins in a sane way is kinda hard, but easy compared to the real problem:

Ensuring ACID (in the case of "traditional" databases). I.e. Atomicity, Consistency, Isolation, Durability.

You need to protect against data corruption in the event of failure, all while guaranteeing atomic operations at the user level concurrently (in most production DBs; SQLite is a notable exception in that it fully serializes writes --- but it can get away with this because it's an embedded database with the primary use case of a single-process writer). And the entire thing must land on a known good state at the end of all of those concurrent transactions.

... and they must do it all while maintaining good performance, and sometimes on a combination of filesystem + hardware that's actively hostile towards the idea of data integrity (e.g. hidden RAM caches in disk or RAID controllers that don't flush on power loss --- thankfully, those are getting rarer, or so I've come to understand).