[gaius]

If you need 1 engineer available 24/7, you need 3 teams of 3, one each in APAC, AMER and EMEA, at minimum. You can't sustain running with less, otherwise people can't ever take vacations or get sick... if you need 2 you need at least 12 and so on.

[peterwwillis]

Apparently you have never worked in a NOC?

You need one team of three for 24/7 NOC coverage, even for a worldwide customer base. Preferably four or more but three will do for a while.

Tickets are automatically generated from alerts for anything from unusual web server logs to disks filling up to network bandwidth spikes to slow SQL queries, etc. The NOC staffer attempts to resolve problems on their own, and identifies and calls the first point of contact for specific issues as necessary, following the appropriate escalation procedure.

As people get woken up in the middle of the night for their shit breaking, after a few weeks, suddenly regular problems either go away or standard procedures for fixes are documented. In a few months a staff of three can handle 90% or more of the tickets that are automatically generated.

We were doing this over 10 years ago with about 2000 servers getting traffic from 2,000 to 150,000 hits per second on Perl web apps. Honestly, nobody does anything efficiently anymore.

[gaius]

3 people who never get sick, never go on vacation or training, who never experience denial of physical access (or who actually live inside the NOC??)

Oh, you are talking about people who just phone other people, well you need to include the "other people" in your number.

[peterwwillis]

The "other people" are just people in your company. They aren't on call, but since they own the thing that's broken they get called, it's standard escalation.

Two people on shift per weekday, three on weekends, and a non-NOC staff member works a shift on a weekend so they can cover sick time and vacation people. It's not that hard.

[gaius]

Your people are doing 12-hour shifts every day, and only get sick at weekends?? I call shenanigans.

[peterwwillis]

They don't need to work during the day on weekdays.

[user5994461]

I am pretty sure that people are not legally allowed to work 8 hours per day, for 7 days per week every week ;)

[peterwwillis]

In the United States you can work pretty much indefinitely as long as there aren't specific regulations against it like for truck drivers.

Part time NOC gets six to twelve hour shifts as needed, and you don't need them 24 hours, you only need them during non office hours.

Weekends are the only time they're there 24 hours, and usually one engineer, developer, devops, *admin, manager, etc rotates out a shift so they have to be familiar with how the system works in case someone gets sick or takes vacation.

[ci5er]

I know it's pushing it, but couldn't one go w/ 1 team of 5 (probably 6 for minimal downtime/buffer?) in NA to cover 24/7?

[Tushon]

Making people operate against circadian rhythm is a hard sell. Google advocates against that in SRE book.

[crdb]

I once visited a Ford factory in England where the manager told us the night shift workers were paid very well (more than London front office in their first two years at Goldman Sachs) because their life expectancy might be shortened by as much as 20 years.

[ci5er]

Ah. I did not know that.

When I was growing up, there was a 24/7 coal-mine near-by and everyone was on two-week rotations for their shift. Which (naturally) sucked. Eternal jet-lag. I thought that letting people simple pick their preferred, and stick to it would solve the suckiness...