Hacker News new | ask | show | jobs
by avinassh 2052 days ago
> AWS has sqs & kinesis which are much better queueing options in that scenario where you’re in AWS doing new dev and can pick a technology.

Why are they better?

2 comments

They aren't. They're technically more correct but not always the practical best choice.

RabbitMQ is a smart play as Rabbit is very easy to use, understand, and troubleshoot at the low end (which is where I suspect the vast majority of queue systems live).

It also has a feature which is actually really hard to do (and sqs doesn't do). Guaranteed delivery of a message once.

That was THE reason we never migrated to SQS, there are scenarios where SQS can double deliver. Our codebase was built up from nothing over time and couldn't gracefully handle double delivery of messages in all scenarios. We could have refactored, but it wasn't worth the work when we were already doing a half billion in revenue without getting even close to the limitations of rabbit AND were close to selling (which we ultimately did).

AWS is great at selling multiple slight variations of the same product. If you look you can usually find ONE variation that works for you. The real test will be if the billing isn't garbage (garbage billing is why we didn't use their other AMQP service and part of the reason why we don't use things like EKS or Managed SFTP despite having the need).

> Guaranteed delivery of a message once

That flies in the face of my distributed systems knowledge. It's not possible in some failure cases.

If your acknowledgement of a message gets lost (because either server involved or the pipes in-between fail) you've processed the message already but the queue server will think you haven't. It either has to resend it (duplicate delivery) or it ignores acknowledgements all together (drops messages that it sent you, but you didn't process - maybe because your server failed.) So the choice when there is a failure in the system is between at least once or at most once - exactly once cannot be guaranteed.

I'm not aware of any way around that predicament.

You are correct, a better description is that their path to 'deliver exactly once to the best of your ability' is clearer.

If I remember correctly SQS is hard limited to a fairly short timeout to requeue messages delivered but not acked. In rabbit it's much more configurable.

Also regular rabbit hosts support the kludge pattern of, 'just run one host and accept if it goes poof you can lose messages,' which is useful if you don't want to bother with the complexity of clustering or are on a shoe string budget.

Lastly you get a nice user interface with the management plugin and you can stand it up locally with docker compose (without depending on AWS for dev or any of the 'aws but on your laptop' solutions).

Yeah, those are nice features to have. Plus you don't get the platform lock in.
Normally I don't give a crap about vendor lock-in, but this would be an exception where I'd agree with you.

Though most people are just going to use a framework plugin to manage the messaging layer, so what's behind that is largely irrelevant.

SQS also supports FIFO queues, which have once-only delivery and ordering. Any reason those didn't work for you?
Aren't they expensive with performance limitations?

Yes we could do that, but we had already been using rabbit in a bunch of places. It made no sense to change it.

They’ve been around for quite some time so they have a much wider customer base & bigger teams supporting them. AWS services can be a bit choppy in the beginning so imo, especially with queues, I’d wait for it to bake.
I agree. SQS has done nothing but improve over time.