[sergei]

1. Yes, I recognize that MongoDB will automatically fail over when we go from N nodes in the set to N - 1. But how do I get back to N nodes? That's completely manual.

2. What happens when I read an update that succeeded on the master but then later fails on the slaves?

[knbanker]

1. It depends on how the node fails. If there's just a network partition, then you still have N nodes, so no issues. If you're running with durability enabled, and you experience, say, a power outage, then the member should rejoin the set and resync with no issues. If a node's drive crashes, then you'll need to restore from a recent snapshot (within a day or so) or perform a complete resync if you don't have snapshot. But this can all be done without taking the replica set offline. In that last case, there is some manual work involved. But your post, unless you've corrected it, implies that replica set failover is completely manual. That's certainly not true.

2. Outside of some kind of hardware failure, you won't have situations where writes succeed on the primary but fail on a secondary. And as I stated on your blog post, if you're really concerned about it, you can specify a write concern on insert, and if the write fails to replicate in the desired way, you'll know about it.

[sergei]

Sorry, but "hardware failure" is a fault, and when you can't deal with it, you're not tolerant. And with larger clusters, you see hardware faults on a regular basis. So saying we're ok in the nominal mode is not fault tolerance.