[Traster]

There's a whole slew of lessons to learn from this. Leaving dead code in your system and then deciding to repurpose it. Manually deploying with no verification. No checks in place to disable a system during crazy behaviour. No real alerting system. No procedures in place for when a system goes wrong. No audit log to refer to when rolling back.

The lesson from this article is kind of funny

>It is not enough to build great software and test it; you also have to ensure it is delivered to market correctly so that your customers get the value you are delivering

While true, I don't see any indication this was great software or that it was properly tested.

>Had Knight implemented an automated deployment system – complete with configuration, deployment and test automation – the error that cause the Knightmare would have been avoided.

Or to put it another way - had Knight implemented a higher quality deployment system than the quality of any of their other systems, they might have avoided this issue.

These stories are never about a single thing gone wrong. The whole point about critical systems is that you should need dozens of things to go wrong for them to fail, and then you should fail safe.

[wikiman]

The fundamental truth of software. Your system is only as good as its worst component. every. single. time.

Deployment is a component. Monitoring is a component. They are also OpEx and therefore "inferior"