I generally don’t mind these disaster stories as there’s usually useful information in there.
Not this one.
1. It isn’t a rite of passage to delete a production database, it’s an extreme accident / fuck up. Stop normalising this. The guy literally deleted the machine without even thinking about the impact.
2. There was no backup. At all. Not even one. Not even one from a week ago. I can’t have any sympathy for someone hosting a production database without any backups running at all. A backup didn’t fail - they literally had purposefully not set up a backup.
3. Using a cloud provider and hosting your entire infrastructure on one virtual machine including your critical persistence layer with no replication. This is so stupid I can’t even begin to feel sympathy.
There’s nothing to be learned from this article other than the fact the author makes it a habit to implement bad practices.
The author didn't just delete the database. He deliberately installed it on an ASG, which is designed to drop instances.
Using an ASG to manage a production database is a tale from the land of What The Everloving Fuck Were You Thinking, as you note, the author makes a habit of bad practices.
It's at least instructive to note that there is only one canonical good practice when working with AWS:
Always assume you are handling a footgun until you have proven you aren't.
For instance, you might think that Cloudformation is a reasonable way to manage a database in RDS because it's an integrated service and you can keep your configuration in source. That must be a good practice, right?
While they have improved the AWS::RDS resource in that there are fewer ways in which small configuration changes will drop your database, there are still big red banners in the documentation.[1]
To do it, you must specify a policy to say, "please don't nuke my prod database."
Because AWS will always default to being a footgun.
I really dislike these "I did dumb things, please give me likes/up votes/virtue points" themed articles. We all know that you're not supposed to get drunk and punch your wife or crash your boss's fork lift doing donuts. When people do those sorts of things about it. Paying the stupid tax because you didn't follow common sense practices and hosed a production database, caused downtime, cost your employer money, fell for a check fraud scheme, got your bike stolen, etc, etc, is not some badge of honor. Why do people tell stories about it as if it is?
> There comes a moment in every person's life when they decide to punch themselves in the face. Really, it's a rite of passage.
> I’m writing this short post to chronicle the events that lead to this dreadful accident along with a few important lessons to learn from this fiasco.
> Before we begin, a quick overview of my fist. I was wearing a knuckle dusters at the time, but this could happen to anyone that has a hand. I decided to start placing my fist into my own face multiple times, with the most force I could.
Jesus Christ I suck at grammar. I meant for that middle sentence to say "When people do those sorts of things they don't brag on the internet about it. "
Not this one.
1. It isn’t a rite of passage to delete a production database, it’s an extreme accident / fuck up. Stop normalising this. The guy literally deleted the machine without even thinking about the impact.
2. There was no backup. At all. Not even one. Not even one from a week ago. I can’t have any sympathy for someone hosting a production database without any backups running at all. A backup didn’t fail - they literally had purposefully not set up a backup.
3. Using a cloud provider and hosting your entire infrastructure on one virtual machine including your critical persistence layer with no replication. This is so stupid I can’t even begin to feel sympathy.
There’s nothing to be learned from this article other than the fact the author makes it a habit to implement bad practices.