[dkjaudyeqooe]

A system that can get itself into a non-functioning state and that can't be supported by an operator or dedicated support person is fundamentally broken and should not be in production. In my view devs should never have access to production, under any circumstances, ever.

This is an artifact of devs (and others) not knowing what they're doing, and just hacking and hoping for the best. It's really not that hard to develop a system that is reliable and supportable in a basic way. Understanding the code shouldn't be a requirement, but understanding the system should, and that's a requirement of support personnel. Put another way, the functional model of the system has to be at a higher level than the code.

[tdeck]

I'd argue that a software system that can be supported by a dedicated operator who isn't a developer is fundamentally broken. Any response protocol that can be handled by someone without familiarity with the system's internal workings can fundamentally be automated. Scaling hardware? Restarting boxes? Ignoring and silencing an alert? Draining traffic to a bad host? These are all fairly simple actions that could theoretically be automated, and have been automated at many companies. There shouldn't be a need for a person who can only do things like this.

On the other hand, there will be production issues caused by a complex interaction within the system that arises in an unforeseen edge case. These issues frequently require a code change which requires the ability to understand the codebase. In that case, the system is broken, but it's not "fundamentally" broken, it's broken for a particular edge case. Unfortunately, we may not have the luxury of waiting until 10 AM PST to start looking into the problem and coming up with a fix for it.