Hacker News new | ask | show | jobs
by timrobinson 5108 days ago
Specifically, it looks like a manual error during a deployment:

http://www.guardian.co.uk/technology/2012/jun/25/how-natwest...

"It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"

1 comments

The aspect that they didn't export the job queue prior to update is something that cross's the border of neglegence with both feet.

Still can't understand why a bank that had people who made bad deals and lost money causing the bank to partialy fail are kept inplace and paid bonus's and the IT people who did there job well are replaced by cheap labour external to the country and this is at a time when there going on about TAX evasion, this too me makes no sence and is why I don't run banks :|.

"who made bad deals and lost money"

Quite an understatement that, the €71bn RBS acquisition of part of ABN Amro often makes it into the lists of "worst business deals ever":

http://www.independent.co.uk/news/business/analysis-and-feat...

IT is seen as an overhead, and as something that doesn't contribute directly to profitability. When compared to dealers, dealers bring in the profit, IT slurps it away.
Very true sadly. Don't see many outsourced HR and accounts sections though.

Sad part is that when it goes wrong it does effect the balance sheet. Even sadder is how they internaly cost IT and do it wrong. Remove the IT and see how many people/time is needed to do the same job and that is the true cost/potentual impact of IT. Sadly though that is never done and only comes to light when things fail and then they blame IT and not the effects of seagul managment, budget cuts etc.

Oh man. Please, please, please outsource HR. Preferably to Mars. Biggest waste of space in the whole company.
Interesting that a retail bank whose core business is to process transactions takes the decision to outsource that capability. And then is exposed as unable to monitor/control the outsourced operation.

Some similarities to the now defunct Railtrack whose remit was to maintain the UK railways decided that it was a good idea to outsource all of its engineering capability. It turned out that Railtrack did not have the ability to manage, monitor or access the state of work carried out by contractors. Some details and links in... http://en.wikipedia.org/wiki/Railtrack#Founding

A good lesson. Accessing which are the core functions of your company. And the ability to execute them without failure.

If it's true that this is a manual error, then I have two questions:

1) Why did it take until the early hours of Wednesday to find out? Why not run a manual check after the change went in?

2) If they found out on Wednesday morning, why did it take until Friday - after three failed batch runs - to fix it?

To both your questions, the answer is lack of expertise at hand. People who knew the smells and idiosyncrasies of the systems were not there anymore. A good engineer (as I'm sure there are plenty in the new offshore teams) can fix mostly anything, he'll just take some time if he never did it before. Now they learned how long those delays were.

Maybe in their next business case for off-shoring, they'll plug in realistic numbers for the risk of downtime and its cost.