Hacker News new | ask | show | jobs
by CaptainOfCoit 233 days ago
I'd add a fourth; "Make it easy to do backups and verify they're correct".

I don't think I've ever considered a data store without that being one of my top concerns. This anxiety comes from real-life experience where the business I worked at had backups enabled for the primary data store for years, but when something finally happened and we lost some production data, we quickly discovered that the backups weren't actually possible to restore from, and had been corrupted this whole time.
2 comments
jjkaczor 233 days ago
Heh - I once made a little chunk of change, because a former client from 10-years previous discovered the shiny "DVD/CD" backups had succumbed to "bit-rot" and needed some source code.

I grabbed the hard-drive off the shelf, put it in an enclosure and handed them the source-code... (At the time, every time I upgraded my system, I would just keep my old drives, so... had a stack of them - buy a new external enclosure, slot it and park it.)

link
navigate8310 233 days ago
Schrödinger's backup. Testing the backup works involves even more engineering and non creative work.
link
CaptainOfCoit 233 days ago
Depends. Even something basic like "Check if the produced artifact is a valid .zip/.tar.gz" can be enough in the beginning, probably would have prevented the issue I shared before.

Then once you grow/need higher reliability, you can start adding more advanced checks, like it has the tables/data structures you expect and so on.

link
6510 233 days ago
I had a funny where I somewhat regularly test an sql backup, then one day it didn't work, it worked the second time, the 3rd and the 4th. I have no idea why it didn't work. It turned into a permanent background process in the back of my head. The endless what-if loop.
link
otterley 233 days ago
I’m not sure what your point is. Business continuity requires a disaster recovery plan that must be tested regularly. It might be considered slog work, but like taking out the garbage, it’s non negotiable and must be done.
link
dessimus 233 days ago
"Great, first you wanted more money to buy compute and storage for dev and staging separate from production, and now you even more for 'testing backups'?!"
link