[wslh]

Last time, a decade ago, I played with RabbitMQ and similar open source queue system they didn't handle congestion and rate limitation well (crashed). From my little experience with classical IBM systems like MQ they knew about that stuff.

[sjdmdlakziggy]

I spent a couple years deploying and integrating IBM MQ, and I own a couple of services currently that interact with it. It was (just a couple years ago) and probably still is more robust than current open source solutions, as long as you weren't planning on forking the code for some very specific use case that requires heavy modification, or the more likely situation of just wanting to save money.

As far as message queues go for something like the federal reserve, IBM MQ is really the only option that wouldn't raise a lot of eyebrows in the industry. The Fed is not the kind of institution that I think people would be lenient on for being open and innovative with their software solutions, banks really need to know that this thing is going to work and they all run IBM MQ themselves already. Not to mention the integration possibilities with Db2 and IMS, which are both huge in finance.

My only complaint is that it is very expensive. That's the story with most of the kind of hardcore technical products IBM sells that are likely to wind up near mainframes. I'd imagine the Fed gets a sizeable discount though.

[MBCook]

Being able to publicly claim your product as the one that underlies part of the federal reserve banking system has got to be worth some real money to IBM.

[sjdmdlakziggy]

That's part of it I'm sure, but it's much more than just advertising. What bank is going to make a risky move to a new message queueing service to save a few bucks when all their connections with the Fed use IBM's? And the hook there goes even deeper, because all of this stuff is deeply integrated with your databases.

[rodgerd]

> My only complaint is that it is very expensive.

On the one hand, I guess. On the other hand, I have a pile of the WebSphere product stack in my life. The app server is ten times the cost of MQ. WPS is another ten times the cost of the app server. It looks cheap when you're buying the really expensive stuff!

Also if you're using MQ for things like guaranteed delivery and making use of clustering and security controls I suspect it's cheaper to pony up for the IBM software than it is to spend time messing about in the thickets of the free alternatives.

[amluto]

There’s always FIX. </snark>

FIX is, at its core, a message queue, and it’s a free financial industry standard. And, for any serious use in which the message queueing ability is important, FIX is wildly unfit for purpose.

Seriously, a deliver-once [0] message queue between two parties is very simple, and it’s truly remarkable how severely it can be screwed up. I’ve never used IBM’s solution, but I assume it actually works.

[0] Yes, deliver-once is impossible. But arranging for a failure to result in all messages up to a point being delivered, subsequent messages not being delivered, and the fact that a failure occurred being obvious to both parties is straightforward.

[pas]

can you please compare this IBM mainframe plus MQ stack with distributed and sufficiently redundant Kafka, for example?

[sjdmdlakziggy]

I can try, but the difficulty there is that Kafka is not really a traditional message queue system and the use cases are different. In short though, redundancy isn't sufficient to solve the reliability issues inherent in financial applications and transaction processing.

If you want througput, Kafka is the better option. Use a message queue for transaction processing against a specific target, where it is a requirement for you to have guaranteed, one-time and one-time only delivery, and possibly that its secret and should only be delivered to one target. You should conceptualize Kafka as being a tool to maintain a sort of distributed, global state. A message queue is more like getting served to go to court.

Technically, Kafka is an event stream, not a message queue (this can be confusing because event streams are often implemented at a surface level like message queues). The difference between an event and a message on a conceptual level is important though. There's plenty of good literature out there if you want to dig deep into the difference.

[pas]

thanks for the reply!

I'm aware that Kafka is low-level (and that there is kmq, which tries to implement a message queue on top of it https://github.com/softwaremill/kmq/ ), but the exactly-once semantics seems isomorphic to having the sender and the receiver doing a 2 phase commit using the log.

what are MQ's guarantees? how are they implemented?