Hacker News new | ask | show | jobs
by Jochim 1283 days ago
I hate separate infrastructure teams with a passion for this reason.

Far too frequently you end up in a situation where someone makes an environment change and blows everything up because they have no understanding of the services they're stewarding.

If you want me to take responsibility, my team should be managing the service end to end.

I feel really strongly against this division of responsibility in software teams. It too often leads to holding up progress and hostile interactions due to each team pursuing their own priorities.

2 comments

> If you want me to take responsibility, my team should be managing the service end to end.

This. I really do not enjoy being called up in the middle of the night to walk a group of people that know absolutely nothing about the system through the steps they need to resolve the issue, because nobody wants to give the “dev” team access to the production environment.

I think the solution that best aligns incentives is the one where the people introducing issues are also the ones called up (and able) to fix them.

Ah, developers empowered to do operations. We should have a catchy name for it... "opsdevs"? :P

Seriously, this is the original idea of the DevOps principals. But they run straight into CIS requirement that "developers do not have access to production code" and the ISO 27001 v2013 requirement of separation of responsibilities. So it'd be great if it happens, it just can't happen in the big B2B spaces.

We allow devs to do things in prod. We are a public company. Sox, Hipaa, ISO27001, GDPR, and all that. Every dev on my team has access to their prod servers and databases (but no access to other team's stuff usually). We deploy multiple times a day. We handle our own oncall. We process billions of individual requests daily for millions of users. We have several thousand employees.

Our compliance requires that all code be reviewed and pass quality assurance before merging and that all prod changes be documented.

That means Dev1 writes the code, the unit and integration tests, sets the right configs in each environment, updates the dashboards for any updated metrics, sets up alerts, and updates runbooks. Dev2 reviews the work, pushes back when any of the above needs more work, and then documents on the jira ticket how they verified stuff. Dev1 or Dev2 merge the code, observe the build, and ensure the code rolls out to prod.

When something goes wrong, the oncall dev on the team is paged and can access all prod systems, and can log in, start and kill things, move files, etc.

All counsel these days, from 'devops' or 'sre' bodies of knowledge is: development and operations are two sides of the same system, they should be integrated better. Companies: got it, create new title/team, in charge of this integration. Seriously?