[timrobinson]

If it's true that this is a manual error, then I have two questions:

1) Why did it take until the early hours of Wednesday to find out? Why not run a manual check after the change went in?

2) If they found out on Wednesday morning, why did it take until Friday - after three failed batch runs - to fix it?

[ovi256]

To both your questions, the answer is lack of expertise at hand. People who knew the smells and idiosyncrasies of the systems were not there anymore. A good engineer (as I'm sure there are plenty in the new offshore teams) can fix mostly anything, he'll just take some time if he never did it before. Now they learned how long those delays were.

Maybe in their next business case for off-shoring, they'll plug in realistic numbers for the risk of downtime and its cost.