[arduinomancer]

I don't really understand this point.

I've found its trivial to just have a unique ID in the message and just check if the ID already exists in a jobs table to avoid double processing.

I pretty much always have some metadata table needed for the job anyway, so its not like I'm building something extra.

[gunapologist99]

jobs table? why not just use the database directly then?

[chris_j]

Use the database directly? I hope you're not suggesting sharing a database between services...

[gunapologist99]

Sharing an SQS queue between services is ok then? Even the most basic example for SQS queues from the AWS SA training course recommends using multiple queues for different classes of video transcoding. All of those benefits evaporate once you share a single queue; if you're just reading that queue to create a job in a database, then the queue provides no benefit.

[chris_j]

I'm not quite clear what you mean by "sharing a single queue between services". If you mean having multiple different services reading from the same queue then no, I wouldn't do that. If you mean having a sending service and a receiving service share the same queue (either directly or indirectly) in the sense that the sender sends a message and the receiver reads it then I can't see any alternative to doing that. If I've misunderstood then would you be able to share what you were thinking of?

I interpret arduinomancer's post above to mean that if the receiving service must action a single message once and only once (which is sometimes but not always necessary) then you need to give each SQS message a unique ID and that ID needs to be stored in as DB by the receiving service. I've used that pattern a few times and it works well (and doesn't result in unnecessary coupling between sender and receiver). In the past, I've used messaging systems that gave stronger transactional guarantees, but those systems were things like JMS or MQSeries and I don't really want to go back to those days.

[gunapologist99]

> I'm not quite clear what you mean by "sharing a single queue between services".

I was referring to your (seemingly snarky, and I apologize if I mistook your intent) comment that you hoped I wasn't suggesting to a shared database between services.

I understand and agree that this is what OP was stating. What I was saying was that having the consumers and producers sharing a queue is not that much different from having them share a database, but a database gives you other guarantees as well. (I am a big fan of queues and have been using SQS since 2007.)

In other words, the database is still going to be your bottleneck/SPOF and a queue just introduces needless complexity; you might as well just write to the DB directly and have a table in the DB (or something like a Redis set) to track the jobs.

A queue with indeterminate ordering is better aimed at idempotent jobs than jobs that will eventually need to be serialized by necessity or design.

[chris_j]

> I was referring to your (seemingly snarky, and I apologize if I mistook your intent) comment that you hoped I wasn't suggesting to a shared database between services.

Apologies, I did write a snarky comment and I'm very sorry that I did that, rather than asking for clarification and/or explaining my position a bit more clearly.

I take the view that sharing an SQS queue (or a topic, or a stream...) between two services is OK. It creates a contract between those two services, of course, and part of that contract might be that there is a uniqueId field that the receiving service is going to have to persist in its own database in order to only action each request once, but it doesn't add an excessive amount of complexity or coupling between the services, not for me at least.

I'm still not OK with having two services share the same database, though, even if you're using it for a narrow purpose. For me, the danger is that you end up introducing more coupling than just agreeing on a message format and messaging semantics. I've been burned by that once or twice and I don't want to get burned by it again and I'm pretty happy that it's widely seen as an anti-pattern these days and personally I dogmatically avoid it. I do take your point that the database is going to end up being a SPOF/bottleneck, though, and I don't really have a great answer for that in the general case.