[WaxProlix]

Oncalls get paged first and then escalate. As they assess impact to other teams and orgs, they usually post their tickets to a shared space. Once multiple team/org impact is determined, leadership and relevant ops groups (networking, eg) get pulled in to a call. A single ticket gets designated the Master Ticket for the Event, and oncalls dump diagnostic info there. Root cause is found (hopefully), affected teams work to mitigate while RC team rushes to fix.

The largest of these calls I've seen was well into the hundreds of sw engineers, managers, network engineers, etc.

[tacLog]

Wow, that makes complete sense for something that is impacting this many people and by extension lots of money.

Thanks for the answer, I have only ever worked with such a small team that we are all on a call every day.

I can imagine it can probably get a little hectic in large group calls? On the engineering side is there a command structure? Like say the root cause was found and RC team is rushing to fix it. But another team wants to mitigate in the mean time in a slightly risky way. Would their manager make a case with leadership? Would the proposed plan just be put out for general comment as a response to that main ticket?

[Spooky23]

It depends. I’ve managed major incidents with hundreds of participants.

Our major incident process generally had a “suit” call with non-technical executives and people who would be coordinating customer triage, outreach, etc. Then we would have a tech bridge where the key stakeholders did their thing.

We used the Federal incident command system as a model. It’s a great reference point to use as an inspiration.

[malkia]

Any guides on the "Federal incident command system" to read from (e.g. without blindly googling for it). Thanks?

[jimmySixDOF]

In addition, you can look into ITIL/ITSM Incident Management plans, they have well developed process structure to work from as a guideline.

I have also seen organizations recommend Kepner Tregoe method training for real time high pressure problem solving based off Nasa Mission Control systems.

[Spooky23]

https://training.fema.gov/emiweb/is/icsresource/jobaids/

Here’s a good place to start

[NordSteve]

https://training.fema.gov/nims/ is a great entry point.

[692]

each company is different, from my experience it would depend on the severity of the fix, and the severity of the issue. the problem would get resolved by any means ie temporary sticky plaster if necessary.

Another team would then assess and analyse the root cause from a company wide perspective and then assess the risks, costs and impact and then make any modifications (possibly redoing the temporary fix, and fixing it properly)

Real issue, a call center main telephony system and one of the management servers kept crashing causing over 1400 call center people to stop working. Temporary fix was to re boot the servers every 4 hours causing minor pain, but the call staff was up and running.

After a whole stupid week of the engineers not being able to find the route cause it was escalated extremely high and our team was brought in and we found the root cause in seconds (literally)The servers was VMs and the engineers hadn't checked the physical ESX server they were hosted on. another VM on the box caused the server to go unstable (ESX not configured correctly).

BAU project set up to audit/ report and fix all the ESX servers in the company for other stupid config issues

[treis]

The person you're responding to is not exactly wrong. But since the users dropped to 0 pretty quickly it's likely that every team with any monitoring at all got paged. At least that's what would happen at the moderately large company I work for.

[WaxProlix]

I'm giving a much broader example of what a large company might do for high impact events. I have no idea what the insides of Roblox look like specifically.

[fragmede]

Not to mention a VP or three. A well-led company is going to have management in the line of fire, so to speak, so an outage of this scale would wake them as well.

[YZF]

So 1-3 people actually figure it out while everyone else gets in the way? There's no way hundreds of engineers, managers, network engineers etc. can get anything actually done as a group, right?

[Morgawr]

Former Google SRE here, I can share my experience although I've never been involved in a large serious outage (thankfully). I've had my fair share of smaller multi-team outages though.

Usually the way it works is so that we have multiple clearly-identified and properly-handed-off roles. There's an Incident Commander (IC) role, whose job is to basically oversee the whole situation, there's various responders (including a primary one) whose job is to mitigate/fix the problems usually relating their own teams/platform/infra (networking, security, virtualization clusters, capacity planning, logging, etc. depends on the outage). There's also sometimes a communication person (I forget the role name specifically) whose job is to keep people updated, both internal to the outage (responders, etc) and outsiders (dealing with public-facing comms, either to other internal teams affected by the outage or even external customers).

Depending on the size of the outage, the IC may establish a specific "war room" channel (used to be an IRC chatroom, not sure what they use these days though) where most communication from various interested parties will take place. The advantage of a chatroom is that it lets you maintain communication logs and timestams (useful for postmortem and timeline purposes), and it helps when handing off to the next oncaller during a shift change (they can read the history of what happened).

> There's no way hundreds of engineers, managers, network engineers etc. can get anything actually done as a group, right?

Most people will not really be doing much but when you need to diagnose a problem, having a lot of brains with various expertise in different domains helps, especially if those people are the ones that have implemented a certain service that might be obscure to the other oncallers. Generally speaking, it wouldn't be unheard of to have 30-40 people in the same irc channel brainstorming and coordinating a cross-team effort to mitigate a problem, but into the hundreds? Not quite sure about that much.

Just my two cents. You can probably get more info by reading the Google SRE book https://sre.google/books/

[YZF]

Yeah, I've read the Google SRE book and the product I work on follows Google's SRE model. Sometimes I wonder though if it's all one big anti-pattern. Maybe more precisely it's a pattern designed to work even if nobody knows what's going on. Things are so vastly (over?) complicated. The original designers are long gone. But you still somehow have to keep things going and address any issues that pop up. In our org that SRE model leads that some very weird things because the SREs know the infrastructure (to some degree) but don't really understand the stuff running over it. But I guess we're delivering the service so that's something.

I think the "real world" doesn't work like that. The way the real world works is that things are decoupled in a way that one system's failure doesn't bring the entire world down. So things can be solved in isolation by people that actually understand the system and/or systems are designed in a way that they are serviceable etc.

When the power fails in my neighbourhood, you don't get 100 engineers on a hotline, one van comes down, troubleshoots the problem, and fixes it. Like 3 technicians.

I know there are some exceptions like some power failures that cascaded or the global supply shortages. But those are design failures IMO. A computer system that goes down for this length of time and nobody can figure out why or recover, that seems like a total failure to me on multiple levels. We're just doing this wrong.

[Morgawr]

Speaking from personal experience, most outages are contained and mitigated within a specific service before they end up impacting other services too. Cascade effects are rare, you just notice them more often because they affect multiple people and usually external-facing customers too. In reality, most things will (or, rather, *should*) page you well before it becomes a cascade-effect incident that multiple teams will have to take care of.

If your problem is that nobody knows what's going on and that stuff constantly brings down a bunch of different systems, you either need to finetune your alerting so the affected system tells you something is wrong *before* it reaches other people (monitor your partial rollouts, canary releases, capacity bursts, etc), or you have a problem with playbooks.

The person that implemented the system doesn't need to be the person that fixes it in case there's a problem. We have playbooks that tell us exactly what to do, where to go, which flags to flip, which machine to bring down/bring up, etc in case of various problems. These should be written by the person that implemented the system and any following SRE who's been in charge of fixing bugs or finding issues as a way for the next SRE oncall to not be lost when navigating that space. Remember that the person oncall is not the one responsible for fixing the issue, they are the person responsible for mitigating the problem until the most appropriate person can fix it (preferably not outside working hours).

Again, there can be exceptions that require multiple engineers to work together on multiple services, but in reality that should not be the norm. Most of the pages I handled as an SRE were "silly" things that were self-contained to our team and our service and our customers never even noticed anything was wrong in the first place.

[WaxProlix]

In a really large company, you're talking maybe ~100-200 people per org. EC2 alone has a massive footprint, for instance. Hundreds of engineers, of whom a dozen are maybe oncall for their respective components. If something goes wrong in, let's say' cloudwatch, but EC2 is impacted, that's dozens of people working to weight their services out of the impacted AZ, change cache settings, bounce fleets, etc.

A lot of the time root cause is solved by a smaller number of people. But identifying root cause and mitigating impact during an event -- and then communicating specifics of that impact -- can fall to a much larger group.

If 1-3 people are actively solving the issue, they do so alone, and give periodic updates to the broader group through a manager or other communication liason.

[michaelt]

3 people to fix the Vital Component That Must Work At All Times.

97 people to check/restart/monitor their team's system, because the Vital Component has never failed before so their graceful recovery code is untested or nonexistent.