Hacker News new | ask | show | jobs
How we improved our MongoDB write throughput with SQS (purpleads.io)
1 points by natanavra 1678 days ago
1 comments

Why are you using AWS SQS but hosting your own MongoDB? i.e. why are you not using MongoDB Atlas?

(I work for MongoDB).

We store a lot of historical data that we periodically go back to, reaching 500GB of storage.

Our current AWS setup costs us ~$500/month, last time I checked it's really expensive hosting with MongoDB Atlas, or any managed mongodb and would reach $3k+.

If you kept the whole 500GB in spinning disks you can blow through a lot of cost for sure. However as the older data is infrequently accessed you could keep that in an online archive which is much cheaper. That archive can be queried seamlessly with your hot data at the cost of additional latency. Online archive stores data in S3 rather than spinning disks, hence the reduced cost.

You should also be aware that running a two node replica-set means if one node fails and you have set a write-concern of majority (which you must do to ensure you do no lose data) then writes will fail at the client if either node is down. So you are saving money at the cost of durability. It's important you be aware of those tradeoffs.

See https://www.mongodb.com/atlas/online-archive for info on archiving and https://docs.mongodb.com/manual/core/replica-set-write-conce... for more info on how write-concerns are used in a replica-set.

The default for write-concerns on Atlas clusters is w:majority. i.e. each write must have been received by a majority of the members in the cluster before the client recieves acknowledgement.

(I work for MongoDB).