[Roodgorf]

> SQS is not an event source :(

I really don't understand why this is the case. Every time we need a queue for something we end up having to explain this again to whoever our newest developers are, which seems to me like a sign of how unintuitive it is.

I have yet to encounter a use case where we didn't want to grab items out of the queue relatively frequently or even as they were added, so why force us to mess with scheduled events and long polling to do so?

[vacri]

SQS is probably carrying design baggage from when it was introduced in 2004. I've never used it, but would Kinesis be better for what you want to do?

[zxcmx]

Ahh kinesis has crazy gotchas depending on what you're doing. If you are pushing LOTS of events it kinda makes sense but see: https://brandur.org/kinesis-in-production for some gory details.

In particular: "you get 5 reads". Number of consumers impacts latency, and not ms of latency, but s of latency.

This stuff is not obvious until you really try and use it.

[Rapzid]

The API has tons of gotchas around pushing lots of events too; particularly from many producers.

New programming test idea for hopeful candidates; create a Kinesis client that reliably publishes batches of messages and doesn't starve other producers.

[Roodgorf]

In most cases I think SNS is closer to what I want: multiple producers feeding into a single pipeline that notifies its consumer as soon as possible. The main thing from SQS is the ability to retry a failed message.

If SQS were event based, you could potentially try to process a message as soon as it was received and then have a batch job to reprocess any messages that have been left in the queue however often you like. The closest to this we got was by using SNS and feeding it into SQS, or even a table somewhere, but that gets sort of messy pretty quickly and didn't seem worth the extra infrastructure.

Edit: I didn't really respond to the suggestion of Kinesis here. I haven't done much with Kinesis, but I get the sense it's overkill for the use cases I'm thinking about. I don't plan to have a large amount of producers and don't expect them to have a consistent or large stream of data. If I'm wrong in that being the main use case for Kinesis please correct me.

[vacri]

I haven't used kinesis myself, I've just heard that it's "SQS with the FIFO issue fixed", and wondered if it might also be suitable for you. Judging from the other comments, it wouldn't be a silver bullet, though.

[vacri]

Speak of the devil - I've just had to create a new SQS queue just now (first time in months), and there's an option to make it FIFO.