These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.
Sure it is fun, but the stakes can be high and the stress gets piled on.
It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off!
Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed.
Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.
I remember stuff like that, once the COO called the entire company to the common room to present me a bottle of champagne for my efforts (another time I was given a fancy bottle of whiskey that some marketing guys drank 3/4 of before I ever got around to it). I rather not work any overtime instead.
I've thought about that before. You get the call. "It's 30 minutes away. I'll be there in 10"
If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.
I don't know of a company that does this sort of work, but I know of some technologies that experts in receive these types of calls. The one that comes to mind is an ERP-focused database system. It's called "Progress" by "OpenEdge". IMHO, it's awful, but this has no hindered adoption in the slightest. I wrote Progress/4gl (their query language) often enough in a prior position to have it on my resume. Every 2 or 3 months, I'll get an email/call, asking if I could be available for short-term contracting upwards of $200/hr for Progress emergencies. I have declined all of these, because I found it soul crushing to work with in the past. However, if you could enjoy that sort of thing, that's one example of a very lucrative field to dabble in.
I've gotten a few pings along similar lines for my HighJump experience (warehouse management system, for those not acquainted). And much like OpenEdge, it's pretty... rough around the edges (at its core it's basically a runtime for a VM that's programmed with a "language" (if you can call it that) driven entirely by conditional GOTOs developed entirely in a half-baked SQL-backed IDE called "Architect"; this is paired with a DB schema from hell for all data storage, and it's filled to the brim with sprocs because even fucking T-SQL is more ergonomic than anything doable in Architect). And yet, it was actually kind of fun (in the twisted, Dwarf-Fortress-esque sense of the word) to hack on that system and abuse the hell out of it.
And for some reason warehouse managers seem to swear by it, so it still gets a decent number of new customers - meaning those customers need implementers. And since it's a giant pile of hacks, the average deployment needs a whole lot of customizations - so more implementation man-hours, and a steady stream of maintenance man-hours. Thus, I get pinged every once in awhile for some long-term contract. Too bad they didn't ping me when I was actually looking for work last year, else I probably would've accepted one for the hell of it. Still tempted to; would be an interesting side job, albeit probably soul crushing.
I've worked such a job at a large enterprise. It really does feel like firefighting (minus all the smoke inhalation and physical strain and death risk).
However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports.
An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching.
But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.
I spoke with a CISO of a large F500, his biggest gripe was that he has a team of 30 that can barely keep their heads above water, let alone respond to incidents
It is... interesting. I have been doing this over last 4 years. The biggest problem, as I pointed out before, is the disconnect between what companies claim they would do to fix the problems vs. what they would actually do. So ~90% of selling is repeatedly explaining the same things to different people whose jobs are in the eminent danger of being restructured/deprioritized/eliminated if your services are successfully engaged.
CEO: We are doing project X. We have engaged so and so to do it. You are to support this.
<multiple layers down, Jack, head of IT in charge of backups>: I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for <blah blah blah blah>
...
CEO: Yes, tell them to do this.
...
Jack, head of IT in charge of backups: Ok, you can have this access.
Consultant: Great. <comes back ten minutes later> It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I could continue?
Jack: This was not in the scope of authorization that I have received. I'm uncomfortable providing this level of access without additional authorization.
Damn that's so true. We hired a network expert to figure latency somewhere between our app and the end user. Spent a month asking autorisations for each gdmn net boxes which were owned by several organizations which were working in the same building but not able to cooperate because they were not paid by the same institution.
The guy did very creative reporting though :-)
And the problem was not fixed, of course.
I tried to help him as much as I could, but at some point, the thing is so intricate that you basically give up.
When you’re in this role your goal and job is to build relationships and help people. That includes Jack. The job is getting done with or without Jacks help. So he can either jump onboard or he can be removed and someone else can be added to give access.
In the end people like Jack benefit nothing from being gatekeepers. Because the third time I have to ask for permissions to be changed - I’m asking for Jack to be removed from the task and someone else to be put in place.
When you are in a firefighting role, your job is to fix the fire, not appease Jack and build relationship with him, the person whose practices created the situation to begin with. Unfortunately, even in those cases the highest level of stakeholder who engaged your firefighting service may not be willing to tell Jack's manager or Jack that he will do what he is told or he will be removed.
Jack's gatekeeping is what keeps Jack employed. The company's ultimate bosses will need to make decision if they want to remove Jack and solve a problem or if they do not care that much that the problem is quickly solved. In the vast majority of cases no one in the company who can fire Jack wants to be seen as a bad guy.
If you are an external consultant (with no prior knowledge of the company's systems/processes/hierarchies) brought in to "fight" a particular fire, how do quickly get ramped up to a level of knowledge where you can analyze the root cause of/recommend a fix for a particular "fire"?
I would say the analysis is 90-99% of the work, based on my experience. Especially the more jacked up systems where cheapest devs found work on the codebase and it's a mess. 90% only comes in when it's so jacked up it requires a lot of refactoring.
This is my ideal job, I think. Charge a big rate to be the consultant you call when everything dies from ransomware or OPs problem or whatever, I calmly fix it to normal standards (as possible), take three weeks off after.
Sure it is fun, but the stakes can be high and the stress gets piled on.
It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off!
Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed.
Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.