[mstump]

Some systems cope with failure better than others. Everything you've said is also true of DB running on top of a uniform linux stack. From my experience (500+ large scale production deployments) this doesn't happen very often.

Does it solve all problems? No. Does it make the world a little better and is it better than monolithic single points of failure? Yes.

[carterehsmith]

>> 500+ large scale production deployments

This needs to be qualified....

Did you deploy a single system 500 times, or 500 different systems? Or some combination thereof.

[mstump]

It's a mix. I'm a consultant that specializes in large scale distributed systems. I have some customers that have >100k production database nodes. I manage probably >50PB of data. I have designed large distributed systems for more than 100 customers.

[user5994461]

consultant = charge > £600 a day to bring Docker to the company. Yet doesn't care when shit hits the fan 3 months later because he's already gone. In fact, he will never known about it.

By the way, How to have 100 customers => leave right after the design phase every single time. Clients add up quickly.

[mstump]

I do a mix of pure consulting but also managed services. I typically have a 12 hour SLA for issues, and 1 hour SLA for some customers. 24/7 support for mission critical, revenue generating systems. So no, I'm not just a talking head. It's usually me in the NOC on the hook in case things go wrong. I'm the world expert in this field, if you want things to work at scale people call me.

[mstump]

Individually I charge several orders of magnitude greater than what you're quoting. I'll advise and design, do deep troubleshooting etc.. Consultants that work for me (I'm the CEO) or a large SI will do the implementation.

[carterehsmith]

Nice. 100K database nodes! Is that like, Facebook or Twitter?

I hope you write about that somewhere.

[mstump]

Very large companies, mostly banks, retailers and telecom. Some industrial IoT and a couple governments.

[user5994461]

Some systems fail more often than others.

[closeparen]

Sure, but if it fails often enough that you need to prepare to deal with failure, then the number of times you invoke CleanupAfterFailure() doesn't matter so much.