| Importance of testing has been told multiple times,even though we all ignore it largely. I vividly remember an incident ( this was in the early 2000's ) we had a couple of Linux machines both remote and local.
The testing and development happens in Local and the code is pushed in a short time over dialup connection. One of my team member had a instance of remote open. He was deleting some cache files in his local , he by mistake typed in rm -rf / instead of rm -rf * in the remote machine ,assuming it was his local development machine. He didn't realize the mistake and finding it was taking much longer than expected he cancelled the command. went home happily. He was using a common login. Though he informed his lead , the tech lead also didn't realize the seriousness of the command that moment. What actually happened was every file which had 777 permission ,got deleted till he pressed Crtl-C. Now some of the files in production server was gone and some were still intact. This led to lots of confusion , the person who issued the command didn't realize his mistake , till the server admin who was about to loose his job for faulty permissions some how convinced the management that someone had issued a wrong command , and that he was somehow not at fault. Our Boss was super irritated when he found out what happened. The person who typed the wrong command , along with his tech lead left the job after a few months on their own. It taught very valuable lessons, Especially not to type rm -rf without checking multiple times. I still remember this incident every time I delete something. |
"Measure twice, cut once."
One of the many uses I have for a mindmap in my daily use (Freemind 0.9.0!), is as a place to write out sysadmin/sql commands in a structured way before executing them.
If I can test it on dev before production - great. If I have to run it directly on production, following this method (religiously!) has saved me from any catastrophic (read unrecoverable) issues.
When you are under pressure it can be easy to say 'I don't have time to write out the commands'.
When touching production servers, you need to spend the extra minute or so to write it out.