[malisper]

I've never used SQS but IMO it seems inferior to Kinesis or Kafka. The two big reasons are that you can't have multiple consumers read from a single queue and once data leaves the queue, it's gone forever. Both Kinesis and Kafka let you have multiple consumers and configure a retention period for your messages.

[appwiz]

You can have millions of consumers read concurrently from a single SQS queue. Messages that are read remain in the queue up to the configured retention period or until a consumer calls DeleteMessage.

Source: I’ve built very high volume services that continue to run production workloads and use SQS as the buffer between components.

[malisper]

> You can have millions of consumers read concurrently from a single SQS queue.

We're using different definitions of "consumer". By consumer, I'm talking about a group of workers that processes the data for one purpose. For example you may have one consumer read from the queue to generate various metrics and a second consumer read from the queue and write to a DB. With vanilla SQS, when you process a message, you need to perform all the tasks simultaneously. With Kinesis and Kafka you can have independent groups of workers (i.e. independent consumers), each performing one of these tasks. Each consumer is able to process the queue at it's own rate. The way Amazon recommends doing this in SQS is to have SNS fan out a single SQS queue to multiple SQS queues. Then you can consume each queue independently[0]. That will multiply your costs by the number of queues you have.

> Messages that are read remain in the queue up to the configured retention period or until a consumer calls DeleteMessage.

I'm talking about retaining a message even if it was successfully processed, on the order of days or weeks. I've used this feature of Kafka before to implement a recovery log. Under normal operation, Kafka writes data to a DB. If the DB goes down, you can quickly recover the last N days of data by going through the data retained in Kafka.

[0] https://forums.aws.amazon.com/message.jspa?messageID=865925

[scarface74]

One producer/multiple consumers is what SNS+ Attributes + subscription filters + SQS is for.

If your database goes down, you have point in time recovery and read replicas that can be promoted as needed.

[CoolGuySteve]

How so? I've run thousands of consumers on SQS for batch jobs and it seems to work.

There's also dead letter queue and retries for messages that aren't properly serviced.

[malisper]

See my reply to appwiz: https://news.ycombinator.com/item?id=22280200