I’ve only met a couple people who say “fire”. That’s a little overwrought. The less dramatic version is “we are too big for heroes” which unpacks into something like this:
You don’t want an organization that relies on one or two or three people to swoop in and save the company every time there’s an emergency. A production issue should look like disaster preparedness. Document, train, drill, prevent.
More concretely, make the important parts as obvious as you can, as similar to production as you can, so anyone can set up a system and try to repro the problem. Config, build, and deploy should be push button (very robust scripts with every cheap sanity and safety check you can manage). No flakiness is okay. Nor obscurity. Everybody should understand how it works.
Document everywhere diagnostic data comes from and goes to. Don’t let your most senior people solve production issues; assign them to people one step down (and throw in some promising mid-level team members for
diversity and longevity). Five why’s every outage and be sure your resolutions always involve making the problem easier to detect next time, not just prevent.
You don’t want an organization that relies on one or two or three people to swoop in and save the company every time there’s an emergency. A production issue should look like disaster preparedness. Document, train, drill, prevent.
More concretely, make the important parts as obvious as you can, as similar to production as you can, so anyone can set up a system and try to repro the problem. Config, build, and deploy should be push button (very robust scripts with every cheap sanity and safety check you can manage). No flakiness is okay. Nor obscurity. Everybody should understand how it works.
Document everywhere diagnostic data comes from and goes to. Don’t let your most senior people solve production issues; assign them to people one step down (and throw in some promising mid-level team members for diversity and longevity). Five why’s every outage and be sure your resolutions always involve making the problem easier to detect next time, not just prevent.