[nemo44x]

Fair enough and you're right, the storage system is awful. Switching to Wiredtiger was smart by them and something like that should have been done far in advance.

The problem with updates in MonogDB is that they are inline and flushed to the filesystem that way. So if you have documents which are growing then the old ones have to be 0'd out and a new one written, which results in a ton of wasted space and an unbelievable amount of thrashing. As you saw, ridiculous amounts of IO time and even with SSD's you get pretty bad SSD wear on your disks.

There's ways around this by pre-padding documents or using buckets if you are pushing into arrays but I can see why you didn't bother. Also, the write lock back then was system level - yikes.

Around 2011 MonogDB was incredibly awful. I will say it has improved manifold since then.