[asuffield]

(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google)

Building reliable services out of unreliable dependencies is a part of what we do. At the lowest level, we're building services out of individual machines that have a relatively high rate of failure, and the same basic principles can be applied at every layer of the stack: make a bunch of copies, and make sure their failure modes are uncorrelated.