[mindingdata]

Others are sharing out of the box solutions.

But I will say that many moons ago when I did actually write stuff for Mongo. The oplog was a god send. You can "tail" the oplog, and get every transaction in near real time. We used this for updating Elasticsearch indexes etc in what is basically realtime, without having to poll or modify existing code at all.

[Ankhers]

Mongo 3.6 introduced something called Change Streams[0] which is basically a safer way to tail the oplog. It is also supposed to work well in a sharded environment.

[0] https://docs.mongodb.com/manual/changeStreams/

[chiaolun]

The oplog is awesome! It provides an immutable record which is really useful - we materialize the oplog directly in Athena to get a time-travelling database for debugging purposes.

[kalendos]

Would you mind sharing the process? I'm curious which flow you use to materialize the oplog in Athena/S3.

[sateesh]

Yes you can tail oplog, but in a sharded setup with multiple clusters this becomes a pain.

[sbr464]

The new change streams api resolved a lot of this, since it allows your to utilize the aggregation framework to subscribe to highly filtered/specific queries. The upcoming 4.0 release is expanded further.