Hacker News new | ask | show | jobs
by jascii 2468 days ago
In the 90's and 00's I was responsible for maybe 20-30 servers, now I am responsible for somewhere around the 2000 mark.. While "oh but you can't do that, that's not Infrastructure as Code!" sounds lame, and your co-worker should learn to be a better communicator, things need to be reproducible when working with more complex systems and that calls for more rigorous review practices. Please consider this: It's our pagers that go off at 3am when your code fails..
1 comments
walshemj 2468 days ago
In my experience ops call the developers when something breaks overnight in any case.
link
jascii 2468 days ago
In our organisation, developers have need nor clearance to access live customer data. This means they stay out of production and we have to deal with breakage.

Usually I can just revert to an earlier version and file a bug report, sometimes I have to cherry pick.

Having developers "fix" things in production under time-stress at 3am without proper code review sounds like extremely poor practice.

link
walshemj 2468 days ago
So you have some one every shift who is familiar with every part of a complex system and the requisite programming languages.

And reverting part way through a monthy telco billing run might not be the best idea.

And in my case they also looked after online services and other systems.

link
peterwwillis 2467 days ago
Preventing calling the devs requires dev to front-load ops. Ops needs triage documents, remediation strategies, architectural and workflow diagrams, dependency charts, distributed tracing, intelligent logging, custom application metrics, and tests they can run during incidents to isolate causes.

The devs can do all of this on their own, teach ops how to use it, and then they'd only be called when it was a code issue. But as a dev, you probably don't know all of the above, so ops has to go to dev and be like, "hey y'all, if you don't want to be called, this is what we need."

And this is what DevOps is intended to fix: get everyone in a room, talk about problems, find solutions among everyone. If your org isn't doing this, you can start the change.

link