Hacker News new | ask | show | jobs
by sahilz79 5060 days ago
This was apparently an infrastructure problem of some sort: http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-...

Infrastructure changes can be notoriously difficult to back out by simply using an "off" switch, particularly if this was some type of a firmware upgrade that impacted all of their production servers. Backing it out at a minimum would require some type of a reboot, which would cause problems with an active trades. It could very well be that they were running an Active-Active environment, they had to go Active-Passive, back out the changes from the passive environment, reboot, and surgically cut over to the passive environment. This could easily take 30 minutes.