[justin_oaks]

I spoke out loudly at a previous employer how it's really dumb to have our database and it's backups under a single AWS account. A single AWS account compromise, account issue (e.g. AWS shuts down account), or a disgruntled employee could result in the business being destroyed.

They took some half-hearted efforts to back up the data, but it was far from ideal.

Back up your data, and do it in different places so a failure in one won't affect any other copy of the data.

[robryan]

Yeah pretty cheap and easy, at least relative to the value of data to most companies to setup a server you have physical control of to go download the backup once per day.

[hot_gril]

It's not exactly cheap to always have a backup that you know works. You have to set it up and test it periodically. Of course, that doesn't mean you shouldn't do it.

[fishnchips]

Very much that. As a former SRE responsible for tape storage, I saw things like regular backups of an error phrase "You have no access to this database". Guess what happened when the team accidentally dropped the database?

Unless you're doing regular restores, you don't have a backup. You have hope. So yes, doing backups in a way that gives you some form of guarantee isn't exactly cheap.

[hot_gril]

Or we do have everything in the backup, but the restore process isn't worked out. Someone is losing a weekend writing hacky scripts, and every SLO is being violated, if we ever have to use it for real.

[fishnchips]

Or you have everything in order but the encryption keys are gone.

[account42]

That case is still a lot better than no backups and having to tell your customers that the data is gone.

[wombatpm]

Make them test their disaster recovery plan. One organization I worked for took three tries before they could even simulate disaster their recovery process.