| HN Mirror

I think if you have a reasonable environment, where on-call feeds back to development (which is what OP is suggesting, more or less), you will absolutely get woken up for networking problems, because there's not really an alternative. Maybe some thresholding to allow for minor problems without alerting, but you know. If it's a big enough problem, someone has to fix it, and it doesn't matter if it's your problem or your dependency's problem, it breaks your service so it's your problem. If it happens a lot, you look for another network to run on.

For 3rd party APIs, if they're not critical, you start to develop kill switches. So yeah, someone has to wake up and handle it, but all they have to do is set the kill switch and go to sleep.

Personally, I did dev and on-call for SMS/Voice verification codes. Most of the time, that's in the nasty corner of it's super critical to the application (users can't use the product if they can't get a verification code) and it depends on 3rd parties that have three nines on a good year. In my case, I got tired of dealing with the disruption and developed automated routing that could manage most providers taking an outage without needing me to take action. Results could be better if a human took notice and action, but it was good enough most of the time, and partial outages were much easier for the automation to handle it.

Even if there's no way to do something like that, at least automation can take care of 3rd party API is failing hard, so mostly return errors quickly without trying the API and only let a small fraction of requests go through to sample if the service came back online. That can keep your servers from getting overwhelmed, as well as drive the alert that helps you wake up, yell at the vendor, and decide if you can go back to sleep while the system takes care of itself.

When on-call is disconnected from development, that's when it gets really miserable. If you can swing a shift-work/follow the sun operator job, that's certainly better than on-call where incidents are common and there's no feedback loop to reduce things. It may well be better even if there is a feedback loop, but the feedback loop in that case requires explicit communication and effort; if I'm on call for my own work, I don't have to tell me to not push shit code right before I leave for the day or go to sleep, I'll get that message from myself right away. If someone else is cleaning up after my messes and doesn't communicate the effects to me, I might never know.