Hacker News new | ask | show | jobs
by rremoncake 5147 days ago
To my coworkers defense the service was very new and we had plans to put in a backup system. Bad timing I guess.
7 comments

This is part of risk management of a company. The company had decided to risk the production setup without a backup at the time. Obviously the downtime of the database is an acceptable risk to them. They could bear the consequence when luck turned against them. It should not be on your shoulder.

People make mistakes all the times. What's important is how they recover and be better. I remember a story where a guy was resigning after screwing up a million dollar venture. His boss told him, you have just learned a million-dollar lesson on the company's dime and you are quitting? The guy stayed on and flourished.

Don't feel bad about this. The decision to launch without backups was a legitimate business gamble. Blaming an engineer for the outcome of a business decision is plain bad management.
"With great power comes great responsibility."

No one is blaming your co-workers - but when your founders take investment money and insist on keeping themselves in charge, they also take on responsibility for the performance of the company and its technology. Just because their legal liability is limited, they aren't off the hook for what we think of them running their product this way.

Blowing the production database is bad, yes - but a LOT of bad days for top shelf programmers start with "rm". It sounds like you owned up to your mistake and put in the late nights to fix it in a way that your bosses didn't...best of luck to you.

You don't have to defend people who made a bad mistake worse. It's nothing personal, competent systems people are in short supply these days.
Do you claim of never having made any mistake?

Last I checked, humans do mistake all the time. Not related to systems, or anything really. It's all about being prepared for those and attempting to make as few as possible.

In order to avoid mistakes, you have to know they exist. So either they didn't know backups were important or they declined to prioritize and implement them in anticipation of their launch.

However, I sense you're taking issue with my "bad mistake worse" comment. I don't think anybody is going to argue that dropping a production database is a real mistake. Mistakes happen, and companies (i.e. management) should work to ensure that all the bases are covered, which weren't in this case.

Only important data is backed up. Only backed up data is important.
Everyone has plans to backup data... it isn't until they actually lose data before they think about a backup. This also means that anyone that actually has backups, has probably lost data in the past...
Never ever plan to backup, just backup.