[Naga]

There's a dimension left out here on the Phoenix disaster: It wasn't really IBM's fault at all.

The Canadian public service is really complex. There are multiple unions with multiple overlapping collective bargaining agreements, where the public service is allocated to different classes. These classes are paid specific rates, with retroactive pay being common for changing classes. The majority of the problems with Phoenix have been employees moving from their classes and being paid the correct amount. It has also adversely affected non-unionized positions.

My understanding is that the Harper government (Prime Minister until 2015), who was responsible for the negotiation and for laying out the requirements, was trying to save a money and not responibly create a pay system. Two major factors that jump out at me:

1) Requirements did not call for training. The system was implemented and IBM was not required to train any operators on how the system functions, which is important because all new staff were hired to run the system and,

2) Due to the need for cost saving measures (the government was trying really hard to balance the budget, as the election was coming up), the previous payroll staff were terminated and a new payroll centre was opened in Miramichi, which is a small town in the middle of nowhere.

So, on top of new software, the government lost all of its institutional knowledge regarding payroll and how things are supposed to work. It's actually hard to say how much of this is IBM's fault and how much is the governments, because the government doesn't know how to fix it. No one knows how Phoenix works and no one knows how it is supposed to work. It's just a big mess with no end in sight.

Could IBM have done a better job? Probably, but garbage in, garbage out.

For further reading, the Auditor-General's report: http://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_01...

[danmaz74]

> There's a dimension left out here on the Phoenix disaster: It wasn't really IBM's fault at all.

I respectfully disagree. IBM management either knew what they were contracting was a recipe for disaster, or they were incompetent. In both cases, as they were supposed to be the billion-dollar expert on the matter, they bear great responsibility for this failure.

And at an industry level, companies that promise the impossible push out of the market more honest ones, and they deserve all the bad PR IBM is getting on this one.

[Spooky23]

Payroll projects are always disasters.

Any vendor will do exactly what is asked, no matter what. IBM, Dell, Accenture, Deloitte, McKinsey, etc. There is no reason why a customer should allow a contractor like this to run roughshod over them -- thats incompetence and bad governance.

I've run big projects with vendors like this. You write good RFPs and hold feet to the fire and they will deliver.

[beachy]

> You write good RFPs and hold feet to the fire and they will deliver.

That's the key bit though. Its not easy to write good RFPs. Unless done to an excruciating level of precision (which they very seldom are), you leave wiggle room which the vendor will naturally take advantage of when things get tough.

[Zigurd]

I've written RFP responses in the telecom industry. "Excruciating" is about right. Thing is, if you are not highly detailed you'll fall between the stools of waterfall and agile. It sounds like this project went astray that way. Not sufficiently spec'ed, everybody gets everything they want, and not agile. You cannot make a sensible kanban board out of change orders.

[Spooky23]

That is the face of the problem.

In the construction business, there’s no need to state that nails get hammered or screws turned, unless there is a very specific application. But they tend to structure projects in a way where there is an incentive for success and punishment for failure. That's not to say that it's perfect, but there are far more $10M+ failed IT projects than failed construction projects.

If you work in this industry, you've seen your share of IT vendors delivering barely literate offshore workers through 2-5 layers of contracting pimps. That simply doesn't happen with pipefitters and others.

In IT, the tenure of the CIO other senior leadership is often is lower than the vendor and the project. The incentive for the CIO is to listen to <insert vendor here> because they may communicate what is happening better AND they may want a job with the vendor later. Without the threat of reprisal, many organizations will make poor decisions (both vendor and customer).

[Bertio]

Not to mention government procurement processes do everything in their power to make it difficult to write a good RFP and the major players are highly skilled at responding to RFPs for the purpose of winning the bid, not necessarily to deliver on the RFP's requirements.

[Spooky23]

I've worked in government and commercial sectors. Big procurement organizations are a pain in any situation. Government adds some wackiness because of transparency and personal liability that procurement officers are living with.

But end of the day, if the people writing technical requirements have the time and know WTF they are talking about, it works. If you hire <vendor a> to write an RFP that <vendor b-y> responds to and <vendor z> evaluates, how could the result not be a shitshow? (Btw, that happened in a Fortune 500 org!)

[koffiezet]

I've seen multiple government projects where the RFP's were actually written by the company eventually winning the bid - what a surprise...

[mgkimsal]

you're both right, but you're a bit more right. IBM should have been pointing out that losing the institutional knowledge may cause problems. they possibly assumed some of their milestones and deliverables would be while working with existing staff. if the existing staff is let go, many of the assumptions in the original estimate are just wrong at that point.

[rgbrenner]

The managers needed IBM to tell them about the concept of institutional knowledge? They need to be told that people with experience have learned their jobs? That new employees need training? This is like Management 101. Just how incompetent are these managers?

[hinkley]

So incompetent they hired someone else to write software only they could possibly know how to write.

There is a degree of management bankruptcy implicit in a large proportion of outsourcing projects. Hiring a vendor to handle billing for your public records dept makes sense, because you can’t afford to maintain a stable of experts on billing.

But a lot of the skill brought in by good contracing firms is in extracting and negotiating requirements. Because your management team has no idea how the work actually gets done.

Think about that. It is cheaper to have some other company come in and learn your problem domain than it is for your company to write down clearly what it is they do and figure out a set of steps to attract employees to do the work for them.

Personally, I think that has Management 101 written all over it in large block letters.

[mgkimsal]

Haven't read the docs, so I don't know what the contracts and proposals states, but regardless of how obvious some things are, putting that in writing is a bit of a CYA.

"We're estimating based on having access and participation from your senior/experiences payroll employees. Substantial changes in personnel will impact the viability of the project."

[itronitron]

The competent managers likely had no say in the matter.

[bobdole1234]

That's a trick question, there are no competent managers in government service for the most part. They either burn out from managing without the ability to recruit competent staff, or they moved elsewhere to make drastically more money.

I'm sitting with a 4x multiple on my old public service job.

[Retra]

They're the ones spending the money. It's not "oh, we're getting paid? Caveat emptor." Everyone involved has a social responsibility, but the ones who are getting paid to be there have more responsibility.

[itronitron]

Yeah, with that price tag IBM could have rolled out multiple federated payroll systems and then iterated on the theme until they had a unified system, but they would not have been able to farm out the development to off-site contractors.

[foliovision]

Exactly. Based on IBM's track record around the world (I know of IBM payroll projects for Canada, Australia, Pennsylvania which have failed while New Zealand and Slovenia have been mentioned in comments), I'd say failing on low ball government payroll projects and then taking the government to the cleaners is strategic at IBM.

IBM have made billions doing so. Bad publicity is just a good start.

The really interesting point is that if this business plan is shown to be strategic (if I were Canada, Australia or Pennsylvania I'd be looking at full discovery of all email records), IBM would then be vulnerable to civil suit for fraud.

[drieddust]

IMHO most of the times such concerns are completely ignored by management.

Customer even gets angry and asking for removal of such risks. Vendors comply by sneeking it in the contract in clever words.

[tonyedgecombe]

I've seen that pattern several times, often enough that it's a bit of a red flag as it seems management know their own weakness but don't want to be exposed to it.

[skgoa]

Yeah, some of these comments here seem very naive.

[drieddust]

As someone who works in same space, I can say this is happening because unrealistic expectations of customers are not challenged by bidding Vendors because they fear losing the project. This in turns happens because customer tries to outsmart vendors by pressing them for reducing the timelines to absolute impossible. If any vendor tries to play by the rules, they are thrown out. Smart vendors respond by crafting the contract smarty. Skipping training is one such example. It was crucial but ....

In the end project is either canceled or goes over budged or barely manage to deliver anything.

[Bertio]

Customers love to be lied to; in a recent set of presentations I was favoring the vendor who was presenting reality but the winner chosen by the business evaluators was the vendor who brought in the glossiest presentation and glossed over the complexities and the frightening part expressed a comfort with ever changing requirements. They were put off by the vendors who boasted about zero change requests and making their timelines but liked the vendor who proposed a fantasy timeline with no promises.

[dv_dt]

Yup, this kind of contracts game is what really needs modern innovation. I think the space is kind of living in an extensions of 70's assumptions with many decades of beauracratic rules added on to try to fix up fundamental problem. For another factor, it would be lower risk to allocate into many smaller projects and sign contracts in phases and sub-systems - but then the funding of the total project becomes more at risk politically, and then there is a higher technical load on the contracting dept to architect the boundaries.

[drieddust]

They are barely able to state the outcome of project so expecting them to do the necessary legwork to break projects into logical subprojects is too much to ask.

No disrespect but we have come to a point where IT have started appearing magical to pointy hairy bosses who think juse because it's software everything is infinitely malleable with no impact on quality, cost, or time.

[dv_dt]

If you don't have an organization that can break it down to a smaller projects then there is a question if doing it as one large project has a higher or lower chance of failure.

[drieddust]

Shouldn't that depend upon complexity of change but who cares. CIO have a short tenure so collecting the bonuses and leaving before house of card collapses works.

It's all about managing the personal bonus and career, who cares for the success 9f project?

[dv_dt]

It depends on complexity, but I think it's more of a situation akin to refactoring or rewriting a complex code base. A total rewrite is always an attractive thought, but a real focus on continuous refactoring in smaller portion at a time is often lower risk, lower cost, and faster in the end - and I suspect it's the same with large bureaucratic processes too. It's also a way to sharpen the skills of the organization in small steps with lower risks for failure (and improving future project steps...).

[foliovision]

A responsible vendor then declines to take the project (smart decision) or executes anyway. We've been in the situation a few times where what started as an attractive project with a reasonable budget turned out to be an underestimate and we barely earned back our salary expenditures on the project.

Those companies did continue to do business with us though and we've normally earned back our losses and more on future projects.

That's what honest vendors do. That's not what IBM did.

[GnarfGnarf]

I concur. What I'm hearing is that the Canadian Federal payroll is extremely complex (like all government payrolls, I suppose). The Harper gov't justified the system by laying off everyone who knew the old payroll system.

The software developed by IBM was so complex, that it required the expertise of the people who got laid off to operate it. The novices who took over didn't know how to make the software work, and chaos ensued.

The Canadian government is as much at fault as the developers on this one. (I speak as one who is paying for this fiasco :o).

[foliovision]

IBM has failed to build these state level systems successfully around the world (Canada, Australia, Pennsylvania, Slovenia, New Zealand). IBM should know how to go about building them right or IBM should not be in this business.

The first $200 million of cost overruns I'm willing to pin on the Government of Canada. The subsequent billion dollars and the failure of the system to work at all is the vendor's fault.

If the Government of Canada is entirely impossible to work with, the vendor should have shut the project down at 200% of budget rather than run it to six times budget and still fail.

[kpil]

I agree that buyers typically deceives themselves due to their limited understanding of highly complex large systems, and that their incompetence is amplified by a sub-optimal procurement process.

It's obvious that there was zero actual knowledge involved related to implementing large systems - just by the fact that someone thought it was a good idea to take a behemoth system excreted out of Oracle, and then pass it through the corporate body of IBM, before inserting it into a huge government organisation, while at the same time getting rid of any process expertise and business knowledge they might actually have on-board.

However - I don't think it's probable that any amount of training could save the situation to any measurable degree whatsoever.

Is IBM to blame? Well, I think they knew what would happen.

[toast0]

At some level, isn't IBM supposed to point out that training and transition are important for a large project and shouldn't be left out?

[otoburb]

They most likely did raise this, but when asked to provide a more "competitive" bid, they axed training and who knows what else. This is common in large enterprise sales deals. For professional services deployment engagements, there is always the possibility of an amendment to the services scope (often referred to as a change order or change request).

If the project is estimated to take over a year to deploy, by then you're in a new budget cycle and the vendor has come to an agreement with the procurement department to charge for additional services as required in subsequent phases.

[LifeLiverTransp]

Another nice thing about governments is that they do not deal in monetary damages but in media fallout- so something loud (collapse of a billion dollar project) is worser to them, then silently going over budget because additional points appear on the list.

So biding yourself into a project to low gives you enough leverage when the point of no return is reached to still make a big plus.

[opportune]

This is why it's not accurate to claim that "it's not IBM's fault." They are fully culpable for making a bid that didn't include necessary components of the project priced in.

[foliovision]

Exactly. Based on their track record, IBM clearly knew what was going to happen and even anticipated the extra income as the project spiralled around the drain.

[coldacid]

Sure, if they don't want to be the winning (read: lowest) bidder. Shit constraints --> shit results.

[CountSessine]

I think if Harper had stayed in office, he would have blamed this on the complicated payroll requirements. He likely would have leaned on PSAC/CUPE to simplify it in the next round of contract negotiations.

[walrus01]

the problems with part #2 have been pretty extensively discussed and detailed already: https://news.ycombinator.com/item?id=16494387

my earlier post: https://news.ycombinator.com/item?id=16495431

basically, it's a political game... the same jobs they "lost" in miramichi because of the firearms registry legal change, they tried to get back. Same pork barrel politics that congressmen in the US play with trying to bring jobs to rural districts.

[Naga]

I missed that article two weeks ago, thanks for linking the discussion.