Hacker News new | ask | show | jobs
by protomyth 3502 days ago
If you're running 24 hours then you should staff for that. If I work my 8 to 10 hours and then have to work another 2 or 3, it tends to wear one down.

Ideally, there would be a financial incentive to fix production bugs so businesses aren't waking up developers. This isn't true when you have exempt employees (US term for no paid overtime) doing your support. I've seen manager go totally friggin stupid with employees who went above and beyond[2].

I still think some managers think we would "code ourselves a mini van"[1] if they paid us for support.

1) http://dilbert.com/strip/1995-11-13

2) https://news.ycombinator.com/item?id=3015969

2 comments

I really don't like the model of throwing support over the wall to some team that didn't write the software. There is no incentive to improve the software under those circumstances, and the people that get paged aren't in the position to fix what's broken.

If you can fix the problem at 3am without thinking, then so can a computer program. Write that computer program.

I remember working at a utility company. I didn't last long (I was the "web master" - oh for the early 2000s), and nor did much of the support staff.

Why? Mainly because we didn't have "authority" to fix things. That had to come from a PM.

It might also have had something to do with the fact that after Ernst & Young came in to do some IT re-structure consulting, we had the situation where:

Support staff were supposed to be on call rotation for pager duty. PMs were not.

PMs were given company cell phones. Support staff were not.

PMs were given company cars. Support staff were not (note that when I say 'support staff' I don't really mean 'help desk' but programming. And this was a government utility where, rightly or wrongly, nearly everyone had a company car).

Someone observed this. Management's response was that the cars and phones gave the PMs "flexibility". Meanwhile support staff were to take calls on their own phones, and if need be drive in to work at 2am in their personal vehicles. PMs were not to be disturbed out of hours.

A little rant-y, but the underlying point? Companies with poor "pager policies" are likely to be problematic all around.

> There is no incentive to improve the software under those circumstances, and the people that get paged aren't in the position to fix what's broken.

My problem is there is no incentive for a company to fix the software if they aren't staffed for support. Also, I see a lot of DevOps people thinking they are the only ones who can fix or diagnose the problems. Often, software developers aren't experienced in networks, databases, or hardware. Thinking that devops is the be all of 24 hour support is a problem.

> If you can fix the problem at 3am without thinking, then so can a computer program. Write that computer program.

I've notice a bit of a link from uncompensated support to allowing time adding fixes to the software.

Frankly, from a hit-by-a-bus problem, if no one but the people who wrote the software can support it, then you really have a problem.

> Frankly, from a hit-by-a-bus problem, if no one but the people who wrote the software can support it, then you really have a problem.

In theory, you have a few people that wrote the software. Who is doing code reviews? If everyone on your team dies in some horrible accident, the software is going to be the least of everyone's concerns.

Vacations are a more reasonable concern. Include vacations in your stated SLO. Support staff tends to want to go on vacation at the Usual Times too (Thanksgiving, Christmas, July 4, whatever.)

> There is no incentive to improve the software under those circumstances, and the people that get paged aren't in the position to fix what's broken.

Sure, one solution is to make developers responsible for support, too. I find it hard to believe that's the best solution.

Personally, I like that solution, but I'm a big fan of "he who breaks it, fixes it".

If I implement a shitty "solution" to a problem and it ends up causing even worse problems then I should be the one to "feel the pain" and it should be my responsibility to fix it/do it over/do it right. Nobody else should suffer because I f--ked up.

> There is no incentive to improve the software under those circumstances

Classy. This comment is 100% exactly the attitude. This is why Step 1 is to find a new job. This poster is punishing you like a parent would a child. Is this the way you want to be looked at and treated? Did you study as hard as you did and spent so much time developing your craft for this?

There are better opportunities out there. 1,000s of them. Avoid the jrockways of the world and you will have a much happier career and life.

You insulted the person without at any point refuting his point about the incentive and ability to respond quickly to outages.
without at any point refuting his point about the incentive

Grandparent post described a way to create incentive. He posited that without it there wasn't incentive, he did not prove - nor even attempt to argue - that it was the only (or best) way to create incentive.

ability to respond quickly to outages

Depending on the nature of a problem and training of the people involved, there is zero reason to assume the programmer of an application is more qualified to triage and remedy an outage than someone specializing in operations. The problem could be in the infrastructure around the app. Even if the underlying problem is something the developer is well suited to fix, they still might not be have the expertise to deliver the best temporary solution to get online ASAP, and stay there until the real fix can be landed in the codebase.

To conclude, parent poster does not need to waste breath articulating why such obviously fallacious reasoning is bullshit.

> ... there is zero reason to assume the programmer of an application is more qualified to triage and remedy an outage than someone specializing in operations.

That's why we've got AWS, Azure, Google Cloud, and so on, right? The programmer gets to deal with only their code and let someone else worry about the rest of the infrastructure.

Unless you're talking about AWS lambda, all of those things still have software infrastructure that need system administrators.
If you're running 24 hours then you should staff for that.

Right, so what's the problem? The original article isn't talking about putting people who aren't paid for pager duty on pager duty. The OP is whining about the mere idea of it and suggesting that if you have that job you should quit.