[aspyct]

We have a new project (~ 6 years now) where we implemented a queue with RabbitMQ to temporarily store business events before they are stored in a database for reporting later.

It's awesome!

It absorbs the peaks, smoothes them out, acts as a buffer for when the database is down for upgrades, and I think over all these years we only had one small issue with it.

10/10 would recommend.

[slooonz]

What the data consistency story around crashes ? Backup/recovery ?

[aspyct]

Sorry, I'm not sure I understand your question. Can you rephrase?

[slooonz]

Presumably you have backups for both the primary database and the message queue (or maybe no backup for the later). If a disaster happens, requiring you to restore backups, how confident are you that your system as a whole is behaving as expected (no events acknowledged but not processed / no events processed twice).

[1oooqooq]

looks like you could have had a blue/green DB setup which would give you one less system to maintain and other benefits that a simple queue don't provide.

also, what do you do when the queue is down?

[aspyct]

I don't think the queue has ever been down in 6 years. It certainly never was a breaking point.

As for the database, yes, we could do blue/green I guess, but it's a big database and it's more cost effective to rely on the queue.

To be honest, I'm not even sure blue/green would be an option given our constraints.