Hacker News new | ask | show | jobs
by Bogdanovich 3899 days ago
You can have large queue sizes (larger than RAM size) and siberite would still consume small amount of resident memory. You basically don't need a separate server with decent amount of memory for it. You can also can get benefit from two-phase reliable fetch - if your client gets disconnected without confirming a message, the message will be served to another client (very convenient if you use amazon spot instances for your workers).
1 comments

Note that this also means that messages can be delivered more than once and/or that the clients need to remember the messages that they processed. In some setups that can be a showstopper.
Reliable fetch is a feature, not a protocol requirement. You can use simple 'get work_queue' command to just get a message, or you can use 'get work_queue/open', 'get work_queue/close' - two phase fetch if you need a reliable fetch. You can also use 'get work_queue/close/open' command to acknowledge previous message and read a new one.
Ok so you can switch between at-most-once and at-least-once guarantees. While nice to have both options in a message queue, my point still stands.

Each of these have trade-offs and the way it is architectured here, in the at-least-once case you will have to either remember all the processed messages or be prepared to process a message multiple times, whatever that means in your specific use-case.