[nemo44x]

What else did your config have?

Did you use RAID on your HD's? How much RAM did you have? What types of CPU's and how many? Were you updating documents often or writing new ones? Did you have compound indexes built properly? What was your latency requirements? Was MongoDB able to saturate your resources? Did you read off of secondaries at all? Did you shard your DB and if so what was your shard key? Was it random? Could you use it for reads too? How much of your data set did you need to possibly use at any given time?

If you had to "upgrade to SSD's" at 100GB you either had amazingly poor provisions, were doing most everything wrong or both. Messages like yours indicate to me a poor user - not a poor database.

[Xorlev]

Our config was pretty standard for a few years ago (~2011-2013). RAID10 7200RPM disks, I believe they were Sandybridge, 32gb RAM. We did a decent number of updates/s.

10gen couldn't find fault with our setup other than our need to do updates. Which is weird, updating a database? Inconceivable. Anwyays, spinning disks apparently couldn't deal with the seeking. Our lock percentage was incredibly high, which makes sense given our "high" write load. It was during this evaluation that we found many of their "critical statistics" were non-deterministic (like the padding factor, rerun and you get all values 0.0-2.0. Probably fixed by now).

We had 1 index: primary. We did no other lookups. MongoDB managed to saturate only the IO subsystem. We did read from secondaries, but that didn't help overly much given that the write percent was high on them too.

On their recommendation we upgraded to SSDs. That dropped our lock percentages and it stayed low. We never made it to sharding because we simply stopped caring about MongoDB. I inherited it and was responsible for it, but ultimately decided to get rid of it in favor of Cassandra.

It's not a "poor user" when you follow their best practices of the time and have a poor experience. It (is|was) a poor database. The storage engine was braindead, their acquisition of WiredTiger admits that. It's at least improving, ever so slowly.

[nemo44x]

Fair enough and you're right, the storage system is awful. Switching to Wiredtiger was smart by them and something like that should have been done far in advance.

The problem with updates in MonogDB is that they are inline and flushed to the filesystem that way. So if you have documents which are growing then the old ones have to be 0'd out and a new one written, which results in a ton of wasted space and an unbelievable amount of thrashing. As you saw, ridiculous amounts of IO time and even with SSD's you get pretty bad SSD wear on your disks.

There's ways around this by pre-padding documents or using buckets if you are pushing into arrays but I can see why you didn't bother. Also, the write lock back then was system level - yikes.

Around 2011 MonogDB was incredibly awful. I will say it has improved manifold since then.