Hacker News new | ask | show | jobs
by cogman10 641 days ago
I agree, but there are a lot of footguns with RMQ. A great example of one is that you'll slow down your cluster by adding more RMQ servers (something that's bit us in the past). Which is a forgivable mistake as most people would expect that more cores == faster RMQ. (For RMQ, that doesn't work because Durable messages need to be replicated to the other nodes in the cluster. More nodes == more replication)

The ideal RMQ cluster has 3 servers and is dedicated to just a few apps.
2 comments
zbentley 641 days ago
That was true, but the more recent Quorum queues provide more traditional scalability tradeoffs (you can set a ceiling on the number of synchronous replication hops that a published, replicated message goes through): https://www.rabbitmq.com/docs/quorum-queues
link
nurettin 641 days ago
The most common elephant foot gun in the room is buggy processes letting queues grow.

RMQ immediately slows down (due to mnesia causing delays) and processes start dropping messages despite having system resources to grow.

link
pas 641 days ago
can you elaborate on the details? I have some memories about running OpenStack where Rabbit "was slow", but we never figured out why. mnesia is the storage layer?
link
nurettin 641 days ago
Yes, it was using mnesia as the storage layer, and if I had a few dozen queues with a few hundred messages each, it caused timeouts in some clients (celery/kombu is an example).

I decided to add expiry policies to each queue so that the system cleans itself from stale messages and that fixed all the message dropping issues.

4.0 Changelogs state that they are switching to a new k/v storage (switching from experimental to default)

link
pas 641 days ago
Thanks for the details!

Yep, similar symptoms. (OpenStack's services are also written in Python, or at least were back then, so probably similar to Celery.) We had regular problems with RMQ restarting. (Unfortunately I can't recall if it was for OOM or just some BEAM timeout.)

A few hundred messages in a few dozen queues seem ... inconsequential. I mean whatever on-disk / in-memory data structure mnesia has should be able to handle ~100K stale messages ... but, well, of course there's a reason they switched to a new storage component :)

link
rhodin 640 days ago
Mnesia is _not_ the storage layer for messages (except for delayed messages).

Mnesia stores vhosts, users, permissions, queue definitions and more. This is being transitioned to Khepri, which improves a lot of things (maybe most importantly netsplits) but not directly message speeds.

link