[fehguy]

Querying before the writes solved a lot of problems. It gets the object in the working RAM set. When doing an update, the database gets LOCKED when the statement hits the server--that means if your document is not in memory, you have to wait while it gets looked up. This was an easy, easy win for us.

Regarding the corruption, I got an "invalido BSON object" or something on repair, which tells me some object was only partially flushed to disk when the DAS went down. The slave actually worked fine for simple lookups by ID, but there was some issue with the index and I was unable to run filters against it. Luckily the huge collections are only accessed via unique identifier, so this wasn't a huge issue.

[danudey]

This seems like the sort of optimization that should be occurring in MongoDB itself - instead of acquiring the lock, loading the record into memory (if it's not already), then making the change and releasing the lock, acquire the lock after the record has been loaded into memory (if it's not already).

Have you spoken with any of the MongoDB developers about why it's currently the way it is, vs. a more efficient update path?

[fehguy]

I think there are some possible timing issues with making that a general behavior in the server. 10gen did make it the default behavior on slaves, where the inserts are controlled by the oplog (http://jira.mongodb.org/browse/SERVER-1646).

For us, our DB abstraction layer made this behavior so simple to add that we didn't make much fuss about it.