[andrewcooke]

weird. they had a problem caused by a series of bugs, yet the word "test" doesn't appear anywhere in that page.

[pja]

Testing distributed systems is much, much harder than doing so on a monolithic codebase. The number of failure modes goes up very rapidly with the number of nodes in the system & your code has to (in principle) cope with every possible one.

[andrewcooke]

true, but it sounds like they (and perhaps you) have never even heard of the chaos monkey.

[pja]

Randomly killing instances wouldn't have detected this particular failure mode as far as I can see, since the error lay in the inability to resurrect a failed process under certain circumstances.