[hintymad]

> Second, preventing or mitigating an incident early (even by just knowing the right feature flag to turn off) can save huge amounts of money: both immediate lost revenue during the incident and future lost revenue from customers who would have pulled their business or refused to sign pending contracts.

Not to be sarcastic but just to offer an observation: in a sufficiently large or bureaucratic organization, preventing an incident from happening can rarely get you any credit or visibility. Such achievement falls into the bucket of "what you're supposed to do". So, those who navigate company dynamics well would rather let the incident happen and then be loud on the follow-up action items. The trick is not to turn an incident into a diaster, so it's a dedicate act.

[tormeh]

I learned this early in a conservative org. Preventing things is risky. Just keep the solution ready for when things go wrong because then you'll get approval.

[justonceokay]

Anecdotally I know of an engineer in the Excel team. They would keep around a list of low priority security bugs. When they wanted to do improvements on the system they would attach it to one of the security bugs “nearby“ because the change would become approved much more easily than just fixing the problem itself.

[sergeym]

I've had this in my career also, I've had a solution that was deemed too risky to release by management but would have prevented an outage, but when an outage actually happened that was the first thing they wanted to try and it worked gloriously. I'm thinking that if it was released prior to the incident it would have not have had the same impact on my career.

[cloche]

Never waste the opportunity a good crisis presents

[nitwit005]

The examples of high impact all seem like things unlikely to receive recognition.

If you save a sales deal, they'll cheer the sales staff. And pay them a commission, which you will receive no part of.

[skmurphy]

And start to build a relationship with sales that, at least in a B2B firm, can be of significant benefit.

[CobrastanJorji]

Also, disaster is a good signal to the higher ups that there are problems in your org. If you keep putting out every fire with heroics, your boss will know (maybe), but his boss's boss's boss will see that your org is doing great and everything is all code green.

If you let a few things burn down, your boss's boss's boss will notice the fire, and things may improve. It's perhaps the easiest way you have of communicating with them.

[hintymad]

Yeah, communicating upwards about potential issues and what you're doing about them is essential. When a disaster strikes, you'll look like a sage in the eyes of the higher ups.

[themaninthedark]

What if my boss's boss is the root cause of the fires and he just blames the "team" if his boss ever sees a hint the of issues?

[CobrastanJorji]

Your great grandboss will not look favorably upon your grandboss passing blame downhill. Disasters are (hopefully) always the fault of the person in charge, which is why you'll notice that your bosses are unusually involved with meetings about how to avoid more disasters.

[mjklin]

It’s about what people notice. In town governments I’ve heard of cutting popular programs that will provoke an outcry only to get credit for reinstating them, while possibly smuggling through other actions that are necessary but unpopular.

[johnnycool]

Many carears are made and bonuses paidout with heroesim trick.

[bauldursdev]

Can't solve a problem that doesn't exist yet