[andrem]

Little anecdote:

The Itanium led to one of a more remarkable episodes in my career. Around 2007 we were running a heavy workload on MS SQL Server in a fast growing business.

We faced a lot of outages due to DB overload. Instead of trying to investigate and understand the issue better and optimize the software, some external consultants were brought in and recommended to upgrade the hardware to an Itanium based monster. It was a massive piece of hardware with a price tag close to 7 figures.

The thing went live and performance decreased and issues increased. After a couple of weeks of trying to run on the Itanium we switched back to the old setup and then focused on software improvements.

Long story short - after about 8 weeks of dedicated troubleshooting and improvements the whole app became stable, was capable to double workload on the same hardware without outages for the next 12 months.

The Itanium took up a lot of space in the server room before it was dismantled and used as a paperweight. A lawsuit involving multiple parties (supplier,consultant,business) eventually got settled out of court in 2015 (?) long after I left there.

Farewell Itanium :)

[lumost]

It’s funny, I remember many such projects back when my title was “systems engineer”. I swear the appeal was always that company could spend $$ and get a working solution. It after all makes sense that a growing business would need more hardware. But the problem usually was a few terrible queries.

I swear the big benefit of cloud deployments is the teams ability to say “we tried throwing $$ at the problem, if we don’t want to spend $$$ we’ll need to do some work”. And have this convo play out over a day rather than months.

[35fbe7d3d5b9]

> I swear the big benefit of cloud deployments is the teams ability to say “we tried throwing $$ at the problem, if we don’t want to spend $$$ we’ll need to do some work”. And have this convo play out over a day rather than months.

My current job is on the tail end of hypergrowth and we are just starting to get our arms around the years of hacks and inefficiencies that made it possible to succeed. We've had a dozen conversations where we've decided

* to throw $10^2/day at a problem for two weeks so the engineers are free to deliver the features required to land $10^5/year in ARR, then work on perf

* to analyze the system, identify the one or two features that cost the most, and tackle those while leaving the rest alone

* and yes, to translate inefficiencies to real dollars and use that to force prioritization (we do that a lot ;-))

A team that understands cloud computing and can do some cost forecasting makes some amazing things possible.

[api]

> some external consultants were brought in

Is there a single example of that working anywhere ever?

I have never seen one.

[jiggawatts]

I am that "external consultant".

I have recommended buying a huge piece of tin to run SQL Server on as a valid, cost-effective solution to a performance problem. Currently, EPYC CPUs are great value for money, programmers are expensive, and some workloads are too time-consuming to tune.

The customer implemented the change, and it worked.

I have also recommend a reduction in size of a too big SQL Server coupled with some judicious optimisation to reduce the load dramatically. Even expensive programmers can spend a few days of their precious time fixing glaring query issues.

The customer implemented this change also, and it also delivered the promised benefits.

I have the before-and-after metrics to prove that there was a huge benefit in both cases.

In both cases the issues were ongoing, had caused drastic outages, and the internal staff were not capable of resolving the issues on their own.

To be honest, 99% of my job is just to be the outsider that's not playing politics and not stuck in a narrow job description. I'm told to "fix it", so that's what I do. The internal staff have "roles and responsibilities", and they fight with other teams more than they cooperate. Some people actively hate each other. I come in as the neutral party and for a brief shining moment I can get everybody to row in the same direction.

[ChuckMcM]

This: To be honest, 99% of my job is just to be the outsider that's not playing politics and not stuck in a narrow job description. I'm told to "fix it", so that's what I do. The internal staff have "roles and responsibilities", and they fight with other teams more than they cooperate. Some people actively hate each other. I come in as the neutral party and for a brief shining moment I can get everybody to row in the same direction.

That is the single reason driving external consultant hire in many enterprises.

[ngold]

My pops did a consulting gig in the 90s for Rockwell, and he described it as being the sheriff dressed in black.

Always thought that was funny, and probably pretty accurate.

[mprovost]

There was that time a bunch of Silicon Valley engineers went and fixed the healthcare.gov (Obamacare) site for the US govt after the original contractors did a terrible job. But then the first team were already contractors, so maybe the lesson there is, if your first external consultants don't work, just keep bringing in more?

[dekhn]

the engineers who "fixed" the site all worked themselves to near-death "fixing" the site. Not sure that's a real win.

[thehours]

I didn’t know much about this from the contractors’ perspective. This Atlantic article [1] seems like a decent overview for anyone else interested.

[1] https://web.archive.org/web/20210705021438if_/https://www.th...

[luma]

Loads, but successful consulting engagements don't lead to headlines. Source: 25 years as a consultant with many successful customer projects under my belt and zero that landed my name in the press.

[andrem]

Totally agree with this. External consultants are easily blamed. But the problem usually lies elsewhere - someone makes a decision without understanding the full scope of the challenge.

[ehvatum]

Yes, it works when your best people get tired of middle management interference and quit to form a consultancy that upper management eventually hire in desperation.

[Kye]

Some of the best anecdotes on HN are of the genre "company brought me back as a consultant for a multiple of what I made as an employee when they realized their mistake."

[dredmorbius]

Selection bias. The bad cases are the ones you hear about --- lawsuits and politicking (office or government) resulting in airing grievances to the press.

[fatbird]

Yes, this. I've never seen a "crisis averted by timely application of consultants!" post on Medium, but I'm sure many such tales exist, as I'm one of those consultants.