[lallouz]

I would love to see some reflection on this story from OP. What do you think you learned from this experience? Do you think your response was appropriate? What would you have done differently? Are you forever afraid of Prod env?

Many, many , many of us have been in this situation before, whether as 'monumental' or not. So it is interesting to hear how others handle it.

[mkrecny]

OP here.

I realize that the dev environment was a recipe for disaster, and I was simply the one to step on the mine .. but I believe my guilt about leaving the company is 'quite right'. Thankfully I'm not forever afraid of Prod env - I still do a lot of risky stuff .. but I always have nightly backups, and other 'recreate the data' strategies in place.

[Morendil]

See this: http://www.slideshare.net/danmil30/how-to-run-a-5-whys-with-...

Guilt is a moral concept; when it comes to a run-of-the-mill operations mistake like yours, it does not belong in analysis of its consequences to the business. You are not a robot. You have made and will make mistakes this bad and worse.

Only consequentialist thinking should be the order of the day here; "what do we know that can prevent a similar mistake from hurting our bottom line". In this case backups are the standard, reasonable, well-known practice. Nothing will be improved by a firing or a resignation, by blaming or by shaming.

Insofar as the real root cause of the problem was not addressed, it's a reasonable prediction that any such company eventually joins the deadpool due to similar oversights.

[Kesty]

Everyone makes, has made and will make mistakes. Junior/Senior is not important.

You also could set up 20 layers of dev environments and it still doesn't matter, mistakes can still reach the outer layer.

You need to have the ability to recover from any problem quickly and with the data as updated as you need it to be.

[konstruktor]

Risk avoidance (decent staging) and risk mitigation (backups) are two mostly orthogonal aspects of risk management. Often, a backup will be a good first step for a totally messed up system. However, saying that mistakes will aways reach the outer layer to discount the value of risk avoidance is talking about the possibility of risk realisation where what matters is probability.

[Kesty]

I'm not trying to discount the value of risk avoidance they are both important and should both be used, but mitigation should always be the priority of the two.

1) When you have neither you should focus on risk mitigation first. 2) Having a great and complex risk avoidance policy in place is a good thing but doesn't mean that you need a lesser mitigation system.