[4e530344963049]

If you have been that person before, it can be fun and exciting to think up solutions and implement them under pressure. Definitely a real test in your ability to concentrate. But don't expect the credit you deserve for it, and don't even expect others to even bother to put in any effort towards fixing things.

Also, make sure everyone knows you are going on vacation once you resolve things.

[paulz_]

These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.

[mattlondon]

Constant "emergencies" soon wear you down.

Sure it is fun, but the stakes can be high and the stress gets piled on.

It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off!

Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed.

Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.

[ido]

I remember stuff like that, once the COO called the entire company to the common room to present me a bottle of champagne for my efforts (another time I was given a fancy bottle of whiskey that some marketing guys drank 3/4 of before I ever got around to it). I rather not work any overtime instead.

[Aeolun]

Come work for our company! We have an infinite amount of fires to put out.

To be fair, the number of fires is finite, new ones are just getting started faster than they are put out.

[sanedigital]

I believe that implies that the fires are indeed infinite, just countably so.

[4e530344963049]

A "firefighting" consultancy might be fun, no? And you would give people a lot of time off!

[paulz_]

I've thought about that before. You get the call. "It's 30 minutes away. I'll be there in 10"

If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.

[clipradiowallet]

I don't know of a company that does this sort of work, but I know of some technologies that experts in receive these types of calls. The one that comes to mind is an ERP-focused database system. It's called "Progress" by "OpenEdge". IMHO, it's awful, but this has no hindered adoption in the slightest. I wrote Progress/4gl (their query language) often enough in a prior position to have it on my resume. Every 2 or 3 months, I'll get an email/call, asking if I could be available for short-term contracting upwards of $200/hr for Progress emergencies. I have declined all of these, because I found it soul crushing to work with in the past. However, if you could enjoy that sort of thing, that's one example of a very lucrative field to dabble in.

[yellowapple]

I've gotten a few pings along similar lines for my HighJump experience (warehouse management system, for those not acquainted). And much like OpenEdge, it's pretty... rough around the edges (at its core it's basically a runtime for a VM that's programmed with a "language" (if you can call it that) driven entirely by conditional GOTOs developed entirely in a half-baked SQL-backed IDE called "Architect"; this is paired with a DB schema from hell for all data storage, and it's filled to the brim with sprocs because even fucking T-SQL is more ergonomic than anything doable in Architect). And yet, it was actually kind of fun (in the twisted, Dwarf-Fortress-esque sense of the word) to hack on that system and abuse the hell out of it.

And for some reason warehouse managers seem to swear by it, so it still gets a decent number of new customers - meaning those customers need implementers. And since it's a giant pile of hacks, the average deployment needs a whole lot of customizations - so more implementation man-hours, and a steady stream of maintenance man-hours. Thus, I get pinged every once in awhile for some long-term contract. Too bad they didn't ping me when I was actually looking for work last year, else I probably would've accepted one for the hell of it. Still tempted to; would be an interesting side job, albeit probably soul crushing.

[munchbunny]

Blue teaming in big company cyber security teams will get that for you. Not everything is a true positive but it’s always urgent. Pay is decent too.

[meowface]

I've worked such a job at a large enterprise. It really does feel like firefighting (minus all the smoke inhalation and physical strain and death risk).

However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports.

An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching.

But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.

[htrp]

I spoke with a CISO of a large F500, his biggest gripe was that he has a team of 30 that can barely keep their heads above water, let alone respond to incidents

[insomniacity]

I know Mandiant do this (disaster response) for security incidents. Don't know of a generalised service, but I imagine the big consultancies offer it.

[notyourday]

> A "firefighting" consultancy might be fun, no?

It is... interesting. I have been doing this over last 4 years. The biggest problem, as I pointed out before, is the disconnect between what companies claim they would do to fix the problems vs. what they would actually do. So ~90% of selling is repeatedly explaining the same things to different people whose jobs are in the eminent danger of being restructured/deprioritized/eliminated if your services are successfully engaged.

[beckingz]

hopefully you can skip most of those people and talk to the people who are in danger of being eliminated if your services are NOT engaged?

[notyourday]

CEO: We are doing project X. We have engaged so and so to do it. You are to support this.

<multiple layers down, Jack, head of IT in charge of backups>: I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for <blah blah blah blah>

...

CEO: Yes, tell them to do this.

...

Jack, head of IT in charge of backups: Ok, you can have this access.

Consultant: Great. <comes back ten minutes later> It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I could continue?

Jack: This was not in the scope of authorization that I have received. I'm uncomfortable providing this level of access without additional authorization.

<repeat>

[wiz21c]

Damn that's so true. We hired a network expert to figure latency somewhere between our app and the end user. Spent a month asking autorisations for each gdmn net boxes which were owned by several organizations which were working in the same building but not able to cooperate because they were not paid by the same institution.

The guy did very creative reporting though :-)

And the problem was not fixed, of course.

I tried to help him as much as I could, but at some point, the thing is so intricate that you basically give up.

[digi59404]

This happens, and honestly it happens rarely.

When you’re in this role your goal and job is to build relationships and help people. That includes Jack. The job is getting done with or without Jacks help. So he can either jump onboard or he can be removed and someone else can be added to give access.

In the end people like Jack benefit nothing from being gatekeepers. Because the third time I have to ask for permissions to be changed - I’m asking for Jack to be removed from the task and someone else to be put in place.

[beckingz]

Oh.

[4e530344963049]

Do you mind if I email you with some questions? Thank you.

[notyourday]

Sure. Email is in the profile.

[pratik661]

I always wondered this:

If you are an external consultant (with no prior knowledge of the company's systems/processes/hierarchies) brought in to "fight" a particular fire, how do quickly get ramped up to a level of knowledge where you can analyze the root cause of/recommend a fix for a particular "fire"?

[Clubber]

I would say the analysis is 90-99% of the work, based on my experience. Especially the more jacked up systems where cheapest devs found work on the codebase and it's a mess. 90% only comes in when it's so jacked up it requires a lot of refactoring.

[easton]

This is my ideal job, I think. Charge a big rate to be the consultant you call when everything dies from ransomware or OPs problem or whatever, I calmly fix it to normal standards (as possible), take three weeks off after.

[m463]

It would be fun just to be able to talk to management with the same sort of candor the "cleaner" in pulp fiction did. (harvey keitel)

[papreclip]

Being an SRE is a little more of this and a little less scheduled work. You still have standup and plenty of other meetings, though

[blhack]

This is basically the reason I got out of doing IT and into development.

"If you do things well enough, nobody will think you did anything at all."