[sed_zeppelin]

I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated.

I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign.

There is absolutely no situation in which I will ever participate in another on-call shift. I've been there, I've done it, now that chapter of my life is closed. Find some younger kid, pay them better than you paid me for the miserable intrusion on their life. I'm done.

Just wanted to be the voice who says what, hopefully, some of the more seasoned and battle-scarred readers here are thinking.

[deathanatos]

You expect to not be responsible for what happens to the software you put into production?

(… and I'd like to avoid distracting arguments that amount to "my company does on-call badly" — yeah, those problems do exist and we should strive to fix them. But if I'm to not categorize the argument here as the baby with the bathwater, then we need something to replace on-call with. Prod goes down on a Saturday afternoon; are you going to tell management "tough cookies" until Monday?)

[ggeorgovassilis]

> You expect to not be responsible for what happens to the software you put into production?

First: IT seems to be rather the exception - most professions have no on-call. Eg. even if my car mechanic screws up a service job, they'll have me bring the car back into the garage during their normal working hours, regardless of how and where stranded I am in the middle of the night.

A second comment: I'll be responsible for anything I have created in my own way. The reality of software development is that we implement functional requirements we've been given with which we disagree, we implement non-functional requirements which don't achieve the goal, we are made to use frameworks and tools we're not familiar with, on a short timeline, a low budget and inadequate infrastructure and we're supposed to take responsibility for code our co-workers wrote.

[mikeocool]

> IT seems to be rather the exception

I think there’s actually a fair number of jobs where some level of this is expected.

Doctors are one obvious example — they have on call responsibilities often more onerous than IT, and depending on the situation don’t always receive additional compensation for it.

If you manage people who work different hours from you, in a lot of jobs it’s not uncommon to be called in if shit hits the fan when you’re not working (for example if you’re a hotel manager, to just name one).

I’ve found that any good lawyer I’ve worked with will answer my calls and help me work through things at basically any time of day (their firm might be billing me for the time, but that doesn’t necessarily directly translate to their comp).

Lots of reporters are expected to cover news that breaks on their beat, no matter when it happens.

[quicklime]

> Doctors are one obvious example — they have on call responsibilities often more onerous than IT, and depending on the situation don’t always receive additional compensation for it.

My doctor (primary care physician) doesn’t work outside of business hours. In an emergency the recorded message says to call an ambulance and go to the emergency department at the hospital, which is staffed by a different set of people.

So it seems they do have at least some separation of the oncall aspect?

Lawyers are another story, there’s a lot of things wrong with that profession and we shouldn’t be trying to copy them.

[mikeocool]

If you go to most hospitals at 2AM and need a specialist of some kind (say a specific type of surgeon), there’s going to be someone in that specialty on call whose going to get paged to wake up, come in, and see you.

Even in family practice, it’s not uncommon to be able to get a call back from the on call doctor at the practice on weekends or off hours — if you’ve got a situation that maybe doesn’t warrant the ER, but you’re not sure if it can wait until Monday.

[bsder]

> If you go to most hospitals at 2AM and need a specialist of some kind (say a specific type of surgeon), there’s going to be someone in that specialty on call whose going to get paged to wake up, come in, and see you.

Only if you're dying.

Come in late Friday and you're going to be sitting in a bed until Monday even if your gall bladder is about to explode.

[happymellon]

This is wrong on so many levels.

No they don't.

I know plenty of people who have had to sit around for 8+ hours because the particular type of doctor is not available. The on call only really applies if you're bleeding out.

In my 20+ years of development and support, there has only been once that I was paged due to an actual catastrophic failure. Most are because shitty "SREs" wants monitoring on everything, even if its stuff that I have no control over.

[shafyy]

> Doctors are one obvious example — they have on call responsibilities often more onerous than IT, and depending on the situation don’t always receive additional compensation for it.

I mean.... On call doctors literally save lives. Most on-call software engineers don't. So.

[bloppe]

But think of the shareholders!

[bloppe]

Doctors, firemen and cops are obvious examples, but I've called plumbers at 2am because of a burst pipe flooding the basement. I've called locksmiths well past closing time due to lockout. I've called landlords at all hours for apartment emergencies. Society needs on-callers of all kinds. It's not surprising that some people are vociferously against holding the pager, and I sincerely wish those people success in avoiding it. But someone will always have to step up and they should be appropriately rewarded for it (I've been on-call and was considered lucky to have gotten overtime for it, which I think is strange because it's just a well-aligned incentive structure that any smart company should have)

[al_borland]

My boss recently started an on-call rotation for us. None of the code I have written is customer facing. If everything I wrote breaks at 5:01pm on Friday, external customers will feel 0 impact if I wait to fix it until I show up again on Monday. Worst case, someone internal has to wait to work on something they’ve probably been putting off for months anyway. There are other things they can work on. If it was a constant problem, I’d get it, but a rare instance can be forgiven when no outside impact is felt.

I am responsible for my code, but we need to be realistic about the impact. Not all outages are created equal.

I used to work nights watching over the hardware, operating systems, and applications running in it. We’d do upgrades and break/fix stuff. Some things were worth waking someone up for, but a lot of things weren’t. We’d do what we could do fix it on our own, but for a non-prod environment, it could wait until morning if we couldn’t do it on our own. This idea seems to be lost on people now. I get that 100% uptime of 100% of the systems would be nice, but not at the expense of your employees sanity.

I haven’t actually been called yet with the new rotation, but any week I’m on-call I’m a bit on edge. In the past I had some pretty horrible on-call experiences that pushed me close to quitting, which I won’t get into, so I’m preparing for the worst. I worked my ass off to get into a position where I didn’t need to be on-call and put in my time working nights so other people could sleep. Being back on-call feels like a demotion.

[Retric]

It is a demotion.

[rufus_foreman]

>> You expect to not be responsible for what happens to the software you put into production?

I'm responsible for the software I put into production from 9 AM to 5 PM for about 200 days a year. At 3 AM, I am responsible for taking care of myself by getting a good night's sleep.

If you need 24 hour coverage, taking into account vacations and weekends, you need 5 or 6 people.

[hn_go_brrrrr]

"you need 5-6 people" is moving the goalposts. The root comment said nothing about minimum team size.

[mikedelfino]

If the company has enough people in the team, someone just works the night shifts or on scheduled weekends. No one needs to be on-call because there would be someone taking care of it already.

[nosefurhairdo]

Is the argument here that every software team should have engineers whose normal working hours have 24/7/365 coverage?

[SuperNinKenDo]

If you expect your team to provide 24/7/365 assurance, then it's hard to see how that isn't a perfectly reasonable idea. The only counter to it is that keeping people on call shifts financial cost off the business in the form of psychological cost to its employees. Not very convincing.

[Spooky23]

“Devops” traded less bureaucracy for more accountability.

Have a generalist ops team that is staffed 24x7, or has paid on call as part of the job. They get run books to respond to whatever goes on.

I’ve set this up twice. The first time, we had a team in the Philippines that would cover overnights.

They could start and rollback deployments and do most stuff via the runbook they were provided. Most callouts (5% of escalations) to product teams were due to bad or missing documentation.

The US based team did similar work, just during the day. Both could escalate quality issues for the product team to fix.

The other model was all US, on-call based. We used junior and low-skill folks, who had rotating on-call. They were paid 20% of hourly rate for standby pay and had a minimum pay threshold when they got called. All of that hit the cost center of the offending product or service, so there was both a financial incentive to not get calls, and a human incentive as the engineers didn’t want to get called for escalations. Again, documentation is key.

[Retric]

I don’t put things in production, the company does. And it’s the companies responsible to deal with problems that show up.

24/7 coverage is expensive and mandating someone is on call 24/7 don’t actually provide it.

[joshuamorton]

This is "companies do on-call badly".

For the purposes of this exercise presume that our theoretical on-call process is no worse than Google's SRE structure: You are on-call for a 12 hour shift that is more or less aligned with your waking hours, and you are compensated extra for the time you are on-call outside of normal working hours, whether or not you are called in. You are on-call at most one week per month, on average, and usually less.

[tharkun__]

    You are on-call for a 12 hour shift that is more or less aligned with your waking hours

I suppose if you're Google they can theoretically make it so it's more aligned with your waking hours? Do they do it? Most companies don't or can't. I.e. it's _less_ aligned.

    you are compensated extra for the time you are on-call outside of normal working hours, whether or not you are called in

How much? Way too many on-call processes in which this is nothing but a few dollars to be able to say "see, we do pay for this, even when you're not called!". As in, way not enough for the number being on-call does to how you go about your day. Always on edge, always awaiting that call / alert that requires you to drop whatever you are currently doing. Preventing you from actually doing/starting certain things.

You haven't even mentioned the expected reaction and resolution time and that alone can make a huge difference.

    You are on-call at most one week per month, on average, and usually less.

Great, only one week out of four /s That's crazy if you ask me. Going back to preventing you from going about your day in a normal way. There's no "doing on-call well" in how you describe it.

[ksmith14]

Google staffs SRE teams as either 8 in one location/TZ or two geographically distributed teams of 6 -- often some pairwise combination of U.S., Europe, and Australia to accommodate reasonable on-call shifts.

The on-call compensation varies depending on what tier of service they're offering. Tier 1 (5 minute response time) is 2/3 of your effectively hourly pay for on-call time outside of local business hours and 1/3 for tier 2 (30 min response time). Or time off in lieu.

[joshuamorton]

Note that this is at a minimum, I know some teams with 10-12 folks per location. That just also has downsides since you can end up oncall once a quarter which most people in the role don't like since the extra vacation is nice.

[dheera]

I am capable of writing very good software, testing it, and putting it into production, but I am not capable of being responsible for what happens at 3am on a Sunday. Whether that deal is okay is up to you. I'm okay if you don't want to pick me. There are other jobs I can get. I write good software though.

If the customer is awake at 3am on a Sunday, it's the customer's problem that they were awake at 3am on a Sunday. If it's a social network, I frankly couldn't care; the customer should go to bed. If it's going to be deployed in the emergency room, fine, we should care, but YOU, management, should find people who are actually willing to take that shift (for extra money, or are based in other time zones).

[deprecative]

Why do I give a single shit about the software having an issue at 2am? I don't own the company. I don't care. If they care they can hire night shift triage.

[uuddlrlrbaba]

I expect to be very responsible during my working hours.

[RandomThoughts3]

I have never worked for a company where people building the software and people supporting it when it is critical were the same. The idea is weird to me.

Plus any large enough company should have team in spread out timezones eliminating the need for on call if it’s correctly managed.

[chairmansteve]

Only a disfunctional company would rely on the programmer who wrote the code.

[nosefurhairdo]

I've been the on call engineer on my team for 75+% of the last year (most of my team is contractors, new hire not onboarded to on call rotation yet, etc.).

It's not an issue because we don't break prod. I also feel I'm well compensated. When there have been issues at inconvenient hours, my manager has encouraged me to take it easy after resolving the incident. We've also prioritized improving our integration tests and addressing other issues noted during root cause analysis (RCA), which I suspect is why we haven't had any incidents in recent memory.

If on call duties are this frustrating, I'd argue it's team/organizational dysfunction that is the real problem, and bad on call shifts is just one of the symptoms.

Ultimately, somebody needs to be available to fix a production incident. One person suffering from on call duties is better than thousands of paying customers suffering from broken software.

[alemanek]

On call even if you aren’t actually called is still a burden. No drinking or other impairing substances. Need to be available and ready to help on weekends. So unable to disconnect and go on a hike or some other activity without internet and your laptop.

75% on call even if I was never called would be profoundly unhealthy for me. So I wouldn’t dismiss the toll of just being available 24/7.

EDIT: I forgot to mention I am on an on call rotation but it is 1week on and 7 weeks off. So, not too horrible.

[nosefurhairdo]

This is a good point. I am fortunate to have a good manager who recognizes the unfair burden. I have missed a pagerduty notification before, which my manager dealt with. The incident did not appear to affect my subsequent performance review, as evidenced by top of band compensation.

I would expect stricter accountability with a more reasonable on-call schedule.

[darkwater]

Congrats, you have a very good work ethic but a very poor personal health ethic.

I was like you, and probably still am deep below, and I will fall again in the same trap. But please, try to think about flipping your point of view here and instead of being your manager generous and your company good for not taking into account the time you failed to answer on duty, think about how you are being exploited covering 75% of on duty alone, and the money the company didn't loose just because of you. And how much of that money you got.

[11mariom]

I used to be 50% on-call and it was exhausting. I was just glued to my backpack with laptop in it. It just was BAD.

Even that I had very good manager and if something broke at night I could just disable all the morning alarms and get to office when I wake up… but still. Having all the time in head that you're responsible… nope.

When I stopped being on on-calls it was like HUGE breath to take, and clean the head. I feel a lot better now.

[tengbretson]

This attitude will keep you off the pager rotation, but get used to building meaningless projects or having your expertise relative to the average developer seen as a liability to the org rather than an asset.

[gedy]

I disagree, not the OP but there is a time and place in a career for firefighting, similar to military.

I always take responsibility for my own work, even after hours fixes, etc. But active on-call orgs usually are just reaping tech debt that others sowed. Sorry not going to rally for that.

[0xbadcafebee]

I am 40 yrs old. I get paid a shit-ton of money (just around $200K) to do this stupid tech work job. I work 40 hours a week, I get benefits, flex time, plus I work remote.

If I'm getting paged for a legitimate issue that is related to something I built or maintain, then, yes, I am going to respond on-call. Because it's a fucking privilege to get paid this much money to sit on my ass and type into a screen.

If I'm getting paged repeatedly, or for an issue that isn't my responsibility, then I will get pissed off, and yell and scream until I'm no longer on-call (or they fix the issue, whichever comes first). But I am grateful to be able to have this life. I can spend an hour or two after hours to fix my shit that broke.

[majormajor]

An on-call rotation without sufficient influence over the roadmap and planning to be able to fix persistent problems so they don't repeatedly cause the same issues over and over and over is toxic. And it's gonna kill the team's overall productivity so it's not good for management either. Congrats, you're playing SWE salaries for an ops team that would traditionally cost you less otherwise.

In a more healthy situation an on-call rotation is the price of being able to move quickly, get stuff out the door, and have compensation that reflects that the company isn't paying a whole team of extra people to stare at dashboards 24/7 just for the rare situations that things break after-hours.

Gigs with low-overhead + customers that don't expect 24/7 operations are kinda the real sweet-spot dev compensation + role-wise, but ... pretty rare.

[0xbadcafebee]

Well, I have two thoughts about that:

1) gigs without 24/7 operations are rare, because there is no good reason for a tech product not to be 24/7. it's not costing extra electricity to keep the lights on overnight, nor more staff. there are a bunch of these gigs (my last gig had no customers for 2+ years) but you shouldn't expect them, because part of the reason we're paid so much money is we're expected to deliver "continuous value". most devs would agree with this, because they all want to be able to deploy continuously, whenever they want. (which is a terrible idea, but it is the status quo.) furthermore, if you're doing your job right (and so is Ops), supporting a 24/7 product should not result in on-call pages, because nothing should be breaking outside regular business hours. if it is breaking outside regular hours, somebody sucks at their job. and Ops' job is pretty simple, so...

2) you do have lots of control over the roadmap, planning, etc. but nobody is going to walk up to you and say "hey we were just thinking of maybe doing this in the roadmap, is that okay with you?" you have to get involved, early, and consistently. you have to show you're not going to rock the boat, but that you will have good suggestions, and can show they will turn into better outcomes. you have to play a little politics, a little product ownership, and also an engineering role, in order to influence what the business decides to do. as you get more senior this gets easier because people will defer to you more, but even an extremely likeable junior can influence the roadmap.

on the off-chance that you're just trapped in engineering hell, with hostile management, a terrible product, and a completely apathetic and terrified staff, quit immediately. this isn't normal and you shouldn't think "oh, I'm trapped here." people don't stay in abusive relationships because there's no other choice, they stay because they've justified their own abuse.

[QuarterReptile]

You haven't lived until you've spent a whole weekend at work rushing to fix a production-limiting issue because the boss doesn't know, though you do, about the other division's production-limiting issue which cannot, under any wildly optimistic circumstance, get done in the next two weeks.

Oh, and that weekend is the weekend before Christmas.

[0xbadcafebee]

Oh, I have so, so many on-call stories. The one of "these other people are making our lives miserable" is hard to deal with, but there are paths you can take to get them to work on it. Sometimes it's just not feasible (or is risky) to get them to take more ownership in the short-term. So it's really important to do your own job to establish all the potential failure paths, and set up lines of ownership, make sure your dependencies have their shit together (performance testing, trend analysis, alerts, limits, runbooks, etc) so that when they do inevitably fail you can push back.

I have never been at a job where on-call was done as well as it could be, and most were/are pretty bad in general. But I could always get changes made to on-call, so that when shit started rolling down hill, it didn't hit me.

[renewiltord]

I think the useful thing here is to mention the trade-off. Practically all my SWE friends making $1m+ are instant responders whether denoted so or not.

[throwaway2037]

I know HN doesn't like jokey replies, but you wrote: "all my SWE friends making $1m+", where "friends" is plural. You have multiple friends who earn more than 1M USD total comp per year? Man, HN is getting crazier by the day.

[convolvatron]

on call is like hiring civil and structural engineers to build you a bridge over a canyon, and then when they show up to do a site inspection you just push them in. eventually maybe you'll be able to cross.

[stackskipton]

As SRE, strongly disagree. On Call is like hiring civil and structural engineers then holding them responsible when their poor bridge collapses under the weight of all the traffic.

Sometimes, yes, Devs get called out for stuff outside their control like infrastructure failing. However, at my job, we just had two devs that quit over on call and guess what, their service was one of worst offenders in "Opps, we pushed bug to production."

[convolvatron]

firstly, on call means supporting the entire service. not something in general I built.

secondly, many if not most of the issues that arise are part of some infrastructure automation or third party service or database. expecting me to be fluent in all of those to be useful in the hot seat is a pretty substantial investment and qualifies me to be an SRE on top of my other duties

thirdly, one major reason why my code might fail in production is that it wasn't sufficiently tested, probably because the service as a whole is basically untestable, and even if it were, building test and test infrastructure is likely not at all valued. in many places just filling in that hole would take a year.

onto to the fourth, the story is supposed to be that by operating the service, I'll be incentivized to fix automation and come up with solutions to make it more robust. I actually know how to do this, and every week I'm on call is time that I _dont_ spend doing this. furthermore, getting permission to do so is often like pulling teeth. sounds complicated. sure that would be nice, look at that when you have time in the indefinite future.

so what this often looks like from a development perspective is that I'm being paid to be a developer, I was judged based on my ability to be a developer, but at the end of the day I'm not building the service. I _am_ the service.

[stackskipton]

If you are on call for infrastructure, then I could understand not wanting to be on call. If I'm there, I'm on call for infrastructure as SRE.

I get all political reasons that your code may not work. However, refusing to be on call doesn't fix any of those reasons, it's just ignoring work. Flip side as SRE, I ask if Devs are on call. If they are not, I don't take the job because there is zero incentive for them to fix anything vs churn out 5 features, chuck it over the fence and be like "Ops problem now"

[CoffeeOnWrite]

> but at the end of the day I'm not building the service. I _am_ the service.

I agree. For me though, it gives me pride to own my services and be fully accountable to the business, especially as part of a team with whom I build comradery, and of course our value to the business justifies our good compensation. It only works because we are empowered to make decisions that keep our on calls sustainable.

[badgersnake]

Give them the time and budget to build it like a bridge then. Oh wait, your competitors beat you to market by several years.

Making people work 24/7 is not conducive to good anything, thus on call is a terrible way to do things.

[stackskipton]

On call shouldn't have 24/7 responsibilities. For example, I'm on call and took a call this morning due to MySQL Server running out of space. Terraform change later, it's no longer out of space and I'm back to my day. I'll take 30 minutes it took me to resolve out of my normal time elsewhere.

If on call balloons your 40 hours to 70 hours, yes, you have an issue. That's not normal and you should consider changing jobs.

[Retric]

Waking someone up is going to cost far more than the time spent fixing the issue.

[stackskipton]

I wasn’t woken up. If someone is woken up, it’s expected they take more time off.

[badgersnake]

If you are expected to carry your laptop and answer the phone you are working. Whether you get paged or not is irrelevant.

The corporate gaslighting is strong with this one.

[bdangubic]

you should be looking for another job… immediately… terraforming on YOUR time is no way to live…

[acchow]

In software, building bug-free software is almost never the goal. There is a constant juggling act of tradeoffs between time, requirements, and tech debt.

[ddingus]

I love your comment on this. Perfection.

Once, while traveling in an RV for some work related marketing thing, the discussion turned to the lack of fuel economy...

The RV might perform better if the engine powered the RV by blowing fuel right out the tail pipe. Horrible efficiency, terrible for the planet, and, and all the negatives packed right into a quick expression.

Your comment is on point. Solid and I just felt like sharing my appreciation for the morbid fun it contains.

Nice work. Worth a healthy chuckle. Thanks.

[dheera]

I 100% fully agree with you.

I have survived 2 cardiac arrests (almost died) during high-stress times. I've been stable for a few years now, but only after I enacted VERY HARD boundaries around work/life and never cut down on sleep for any reason (among other health-first changes I made). I have a significant increase in cardiac arrythmias any time I don't sleep enough.

I consider myself at this point as having a disability that prevents me from overworking, and I absolutely need my employers to respect that and accommodate that.

I can work normal hours, and that's my offer. If you want to pay me less, that's okay, but I'm not doing on-call unless it's business hours only.

If customers give a shit about uptime at 2am then it's management's responsibility to find people in other time zones to deal with it, or pay extra for people who are willing to sacrifice and risk their health for a customer (I won't take that deal though).

[srhtftw]

At last, a voice of reason amid the vulgar crowd.

I've held several management positions where I've carried a pager. At one employer I helped keep trading databases operational in 7 time-zones on 3 continents. At another I helped fix a backup issue on Christmas eve. Helping those customers was a core part of my responsibility. I fully understood that and took great pride in it.

But as a developer I too will never accept another on-call rotation.

Companies which assign on-call duties to developers make the mistake that development, management and operations are different kinds of work which require different environments and skill-sets. Other engineering tasks include testing, documentation, training, and maintenance. At small startups the founders and early employees may do some or all of these but that becomes impractical at larger established businesses.

Engineers should learn and do all these things in the course of their career but not all at the same time unless quality isn't a concern.

My experience at a unicorn a few years ago convinced me companies which assign developers on-call rotation either don't understand or don't care about the quality or sustainability of their business. In that company senior management was replaced by folks from Google and Facebook shortly after I joined. I was moved into a team where I had no role in the design, develop or deployment of its services. I had no say in the hiring or firing of the so-called engineers who rushed failing services into place past a wholly ineffective QA department.

I should have seen the writing on the wall when I began to be pressured by managers and recruiters to rubber-stamp candidates who couldn't pass our coding tests but had spent lots of time on-call. The company's priorities slowly became clearer to me as they grew evermore desperate to live up to their promises. Ultimately I suffered an ischemic attack from the stress of this environment and left the company to focus on my health.

Oh and the company? It let go of most of its engineers a year later and was eventually acquired by competitor for a few hundred million after having raised over a billion dollars.

[11mariom]

100% agree.

But many companies does not understand when I tell them "nice offer, but you should consider someone cheaper for on-call duties, and I can do the rest".

When I was younger I was (kinda) happy to take on-calls, as it was a significant boost in my income. With more experience and better base salary… I simply does not need it and… well - I want to LIVE my life, not WORK (yes, being on call is work, not resting time).

[corytheboyd]

Nobody is rooting for on-call, but yeah I put up with it because I am a young stupid idiot thirty-something who needs to make a lot of money now so that the rest of my life can be Nice Enough. I’d love to be able to cherry-pick jobs like this too, but I am not there yet.

Not trying to dunk on you, I’m honestly glad you get to do this, it must make your life considerably better.

[darthrupert]

I had on-call jobs at one point in my career. It roughly doubled my salary, and I put most of that into assets that rose about 20,000% in 10 years. So that was nice.

The job can be excruciatibg though. Don't do it without proper compensation.