[PaulHoule]

I dunno. I've seen companies go through 4 or 5 different message queue systems and find that none of them work quite right.

12 years or so ago I developed a few systems that used qmail as a message queue and I was pretty happy with that.

Asynchronous processing is a necessary evil, but I think a lot of people underestimate the difficulty. It's one thing to compress a video in the background, but if you have one asynchronous task that spawns a bunch of asynchronous tasks and they spawn asychronous tasks and someday they all come together... Well maybe they come together someday. There's a definite "complexity barrier" you hit when asynchronous applications rapidly become harder to maintain.

There are ways around this, but I've frequently seen MQ-based systems that never get "done".

[JeffJenkins]

I believe this is the problem that Storm was supposed to solve ( https://github.com/nathanmarz/storm ). Or Amazon's SWF ( http://aws.amazon.com/swf/ ). They work in the case where you can define the flow between all of the tasks.