[breakfastduck]

I generally don’t mind these disaster stories as there’s usually useful information in there.

Not this one.

1. It isn’t a rite of passage to delete a production database, it’s an extreme accident / fuck up. Stop normalising this. The guy literally deleted the machine without even thinking about the impact.

2. There was no backup. At all. Not even one. Not even one from a week ago. I can’t have any sympathy for someone hosting a production database without any backups running at all. A backup didn’t fail - they literally had purposefully not set up a backup.

3. Using a cloud provider and hosting your entire infrastructure on one virtual machine including your critical persistence layer with no replication. This is so stupid I can’t even begin to feel sympathy.

There’s nothing to be learned from this article other than the fact the author makes it a habit to implement bad practices.

[ben509]

The author didn't just delete the database. He deliberately installed it on an ASG, which is designed to drop instances.

Using an ASG to manage a production database is a tale from the land of What The Everloving Fuck Were You Thinking, as you note, the author makes a habit of bad practices.

It's at least instructive to note that there is only one canonical good practice when working with AWS:

    Always assume you are handling a footgun until you have proven you aren't.

For instance, you might think that Cloudformation is a reasonable way to manage a database in RDS because it's an integrated service and you can keep your configuration in source. That must be a good practice, right?

While they have improved the AWS::RDS resource in that there are fewer ways in which small configuration changes will drop your database, there are still big red banners in the documentation.[1]

To do it, you must specify a policy to say, "please don't nuke my prod database."

Because AWS will always default to being a footgun.

[1]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...