[knightofmars]

Thanks for the flashbacks of supporting failing services at 3:00 AM. :D Even today, I am amazed at how absolutely terrible this type of decision making was and still is. The idea that you can "save some money" by having your engineers supporting your infrastructure "part time" is always going to cost your more in the long run. Because of the long-term effect of having an unskilled (in the area of question) individual attempting to do the job of a system admin (bad security practices, etc) and the turn-over you end up with due to burn-out.

[karmajunkie]

Whenever I've been in this position, or run teams this way, its been less about saving money and more about making sure developers have to live with their own bad calls on tech. I've worked in places where dev teams would just throw code over a wall with the equivalent of a sticker saying "Works on my machine!" and weren't the ones getting woken up in the middle of the night when that overeager query taps out the memory limits, and its a recipe for headbanging.

All things being equal (i.e. I've got the money to have someone who's role is more ops than dev but still does both) having that person to "own" the production configuration is valuable, but developers still need to be in touch with what their code does in production. Otherwise you eventually end up with the equivalent of a cool interaction design that's damn near impossible to implement on the web (another pet peeve of mine...)

[knightofmars]

There's a fundamental difference between holding engineers accountable for the downstream impact of their technical decisions and making engineers take on the added responsibilities of an additional position without compensating them accordingly for the added responsibility. In an organization that prioritizes stability there is an appropriate balance between engineering and system administration as well as potential overlap given the right boundaries and understanding of job responsibilities. The engineering team will be held accountable by the system administration team and changes will happen because of it.

The inherent failing in this structure is when one of two things happens; one, the system administration team does not have the appropriate channels (and clout) to provide push back against the engineering team; two, the technical teams (both engineering and system administration) don't have the ability to get technical debt payed off properly due to an improperly structured project management process.

Anecdotally, I've been witness to the second issue a number of times. If there isn't an immediate understanding of ROI for a proposed change then it isn't prioritized to be worked on. The thought process is generally along the lines of, "Engineers are an expensive resource, having them working on something that won't make the company money is obviously not the priority."

While some of this is on the engineering leadership as their job is to provide insight into ROI for technical matters there also needs to be a balance where the non-technical leadership trusts the technical leadership to know when to prioritize projects with non-obvious ROI.

[karmajunkie]

Definitely agree with much of that, though I don't think it detracts too much from my main point. Most of the companies I work with don't have the luxury of dedicated ops people, and those that do I still think insulate developers too much from the production environment.

[OpenDrapery]

It feels like QA is an antiquated notion these days. Why is a prod-like test environment too much to ask for? As a developer, if you call me in the middle of the night and tell me we are in crisis mode, my first thought might be "Did anyone test this thing with any kind of rigor at all?"

[karmajunkie]

i blame monolithic microservices...