[PaulHoule]

I've never found message queues that hard to work with but it seems everyone else I've worked with inevitably winds up with a queue that messages go into and never get out of.

I've often thought there is an analogy with broken "real-world" processes where the same thing happens.

[XorNot]

The problem is people leap to message queuing to make things fast, but don't build their recovery system first...but the recovery system is exactly what they need to build first. Whatever you're doing with queuing, you need to have a fallback so your service can "catch up" by pulling down what it thinks it missed.

And generally this system should not use the message queue to just get things resent - because most likely in this situation you still need to be mostly handling new messages rather then old ones when you're stuck trying to catch up.

All of which tends to be a product of just utterly magical thinking from managers and solutions architects as to how anything works, and how likely an outage is (basically guaranteed no matter what anyone says about their product).

[trombone5000]

> ...but the recovery system is exactly what they need to build first.

I can't find the reference, but consistent with that, I remember reading someplace that the number of items in your queue should typically be zero, since queues can't actually make the rest of the system any faster. All they can really do is buffer temporary bursts, or buffer while the rest of the system recovers. And for that you'll definitely need your recovery system working as a prerequisite.

[projektfu]

Message cues with ADHD like me. Add to inbox. Inbox full. Declare bankruptcy. Repeat.

[davewritescode]

Most of the problems with message queues are from the fact that you will eventually get messages that you can't process now but might be able to later.

Depending on your requirements you either stop the state of the whole system or use something like dead-lettering and/or delay-queues + retry count headers to make sure that you can reprocess failed messages without letting them accumulate until you're wasting CPU processing messages that will never work.

The other problems with message queues is that you need to make sure you have alerting on all your queues otherwise you're just waiting for trouble.

[slowmotiony]

We use IBM MQ at work and I wouldn't exactly call it simple.

There's sender channels, receiver channels, xmit queues, local queues, remote queues, dead letter queues, topics, subscriptions and all kinds of other things to learn about. Add that to the crappy IBM tools and their documentation that is full of dead links and you're in for a lot of learning through trial-and-error.

[jsymolon]

> We use IBM MQ at work and I wouldn't exactly call it simple.

2nd'd & 3rd'd ...

You hired a 50 man lumberjack crew to cut down a sapling.

I've been spending a some time to remove IBM MQ out and going to "simpler" queuing solutions. First of all, cost. Guess what, we're not getting anything out of licensing per year to justify the cost.

With improving the end points, we don't need all that complexity. With 5 9's uptime on the network and smarter end points, retrying messages isn't expensive.

As always, YMMV.

[kitd]

IBM MQ is the granddaddy of queuing systems and predates most other queuing systems by decades (IIRC it is (ultimately) based on the iSeries/OS400 queuing system). Consequently, it carries a lot of baggage from that time which large ops teams like for tuning to their needs, but which modern systems have largely ignored or modern environments have made redundant.

[pearjuice]

That's because IBM is in the business of selling complex stuff and promising to make it simple with support contracts.

[commandlinefan]

I suffered under MQ for many years. I never understood what any of it was for, other than the actual queue part (which is what you can get for free from competing products like RabbitMQ, ActiveMQ, Kafka, etc.). And it wasn't because I didn't try to find some documentation...