[mimir]

It sort of baffles me how much engineer time is seemingly spent here designing and running these "gamedays" vs just improving and automating the underlying systems. Don't glorify getting paged, glorify systems that can automatically heal themselves.

I spend a good amount of time doing incident management and reliability work.

Red team/blue team gamedays seems like a waste of time. Either you are so early on your reliability journey that trivial things like "does my database failover" are interesting things to test (in which case just fix it). Or, you're a more experienced team and there's little low hanging reliability fruit left. In the later, gamedays seem unlikely to that closely mimic a real world incident. Since low hanging fruit is gone, all your serious incidents tend to be complex failure interactions between various system components. To resolve them quickly, you simply want all the people with deep context on those systems quickly coming up with and testing out competing hypotheses on what might be wrong. Incident management only really matters in the sense that you want to allow the people with the most system context to focus on fixing the actual system. Serious incident management really only comes into play when the issue is large enough to threaten the company + require coordinated work from many orgs/teams.

My team and I spend most of time thinking about how we can automate any repetitive tasks or failover. In the case something can't be automated, we think about how we can increase the observability of the system, so that future issues can be resolved faster.

[kqr]

If you think of incidents as component failures, and the solution as increasing automation related to getting the faulty component back online again, you're under the old view of system failure. This view works for simpler systems.

More complex systems experience failures due to interactions between fully functioning components. The teams that made them didn't, for one reason or another, foresee that mode of interaction.

These are errors designed deeply into the system, and you can't automate recovery. You need to fix the problem at the cause.

Proper analysis is required, and if a game is what it takes to do that then why not? Additionally, it helps people learn to do that analysis on the fly. That is a crucial skill because those incidents are normal in complex systems. They will happen.