Hacker News new | ask | show | jobs
by otterley 109 days ago
The author is extremely lucky that support was able to find a snapshot for him after he deleted them all. I worked for AWS for many years and was a customer for years before that, and they were almost never able to recover deleted customer data. And this is on purpose: when a customer asks AWS to delete data, they want to assure the customer that it is, in fact, gone. That’s a security promise.

So the fact that they were able to do it for the author is both winning the lottery and frankly a little concerning.

What bothers me more is that the Terraform provider is deleting snapshots that are related to, but not, the database resource itself. Once a snapshot is made, that’s supposed to be decoupled from the database for infrastructure management purposes. That needs to be addressed IMO.

UPDATE: deleting previous automated snapshots on database instance or cluster deletion is default behavior in RDS; that’s not the TF provider’s fault. However, default RDS behavior on deletion is to create a final snapshot of the DB. Makes me wonder if that’s what support helped the author recover. If so, the author didn’t technically need support other than to help locate that snapshot.

And yes this is an object lesson of why human-in-the-loop is still very much needed to check the work of agents that can perform destructive actions.
1 comments
yibers 109 days ago
Having customers delete all their data by mistake and then trying to recover it happens more often then you think. It has become common practice to soft delete at first. Usually 30 days later a hard delete is performed.
link
otterley 109 days ago
Oh, I know it happens. Over the years AWS has added functionality across various services to help prevent accidental deletion, but absent some documented behavior to the contrary, when a customer confirms that data is to be deleted, AWS is supposed to make that data completely inaccessible by anyone, including AWS themselves.

I updated my comment above because I have a theory as to what really happened here, and it doesn’t involve support recovering deleted snapshots.

link