[devonkim]

The primary reasons for multiple consumers in a queue is availability and SLA reasons on a queue as well as for easier horizontal scaling. Otherwise you’ll need to have a queue scheduler type system that can signal or serve out queue locators to idle consumers and you start getting into technical scenarios similar to freakin’ ESBs. At enough scale you already have that setup though for multi region failover purposes sure but the granularity of queue consumer routing is based not around concurrency to the queues as much as concurrency and routing across several regions with n queues in between that serve as priority queues.

Also, two different queues being two different buffers that have durability issues can in an improperly conceived architecture amount to a distributed RAID0 of messages.

It really depends upon the tolerance to message duplication, SLA needs, and how prioritization should be handled. At a previous place we had multiple consumers for multiple SQS queues representing different priorities within the same region and it worked fine for many years with the primary headache being message de duplication handling being tricky.

[015a]

This discussion is, at least it seems, mostly about multiple heterogenous consumers; not homogenous consumers/replicas/horizontal scaling. So, if Slack sends a queue message for every DM that's sent, the difference between having 1 consumer that updates the database and 1 consumer that sends a push notification, versus having 2 consumers that both only update the database.

The idea of having multiple homogenous consumers shouldn't be controversial; that's just horizontal scaling. And, well, at least until a few hours ago I also would have said that the idea of having multiple heterogenous consumers is also uncontroversially bad. But I guess everyone has "their way" of doing things.

Its also important to note that there's a third situation I see somewhat often: maybe call it homogenous delegated consumers, whereby you've got messages like '{"type":"SendDM", "content": {}}'. Or maybe: '{"type":"SendDM", "action": "UpdateDB", "content":{}}'. The consumers are still homogenous, they all run the same code, but they may internally delegate the message to do different things depending on enums within the message. This is pretty ok; its different because at least you'd never have a consumer hit message and be like "I don't want this take it back".

Though I'd caution against it; just understand that its something of a 'hack' to make one queue act like N queues, and that's ok if you're small and have a good grasp on the problem domain. The big issue it will inevitably run into is: some queue message "kinds" will take a lot longer to process than others; and so if you're e.g. overloading a queue to handle both a simple email send and a much more complex asynchronous database update, you'll inevitably get delayed emails. Absolutely inevitable. But, it can work for a time.