[estrabd]

This was my take away question. Redis can be used extremely effectively as a pool of job queues with failover. Perhaps RabbitMQ provides robust bidirectional messaging? While pooling Redis works well for one-way job submission (with each Redis instance being backed by some set of work consumers), making the process synchronous (whereby the consuming worker communicates back to the producer) is not so clearly handled in a robust way unless the producer is listening on some set of Redis instances for the single reply message. RabbitMQ seems heavy weight just to solve that single problem, though.

[saryant]

Making something synchronous when it involves a job queue sounds like a recipe for disaster, IMO. Better to let both the consumer and producer act in a fire-and-forget manner with the original consumer producing a reply on a second queue which the original producer will eventually handle.

[estrabd]

Our use case necessitated a synchronous interaction, but the message back was done exactly as you suggest - just uses a private second queue. The challenge was that while the initial job is fire-and-forget (or wait for a reply), the 2nd private queue was just a placeholder for a reliable messaging backend that needed to be implemented. It doesn't matter what worker gets the job, but it matters to whom the consumer of the job replies. I believe Redis provides enough primitives to make a highly reliable messaging system. We just have not done it.