[ritwikt]

Would be nice to know why RabbitMQ/ZeroMQ didn't fit the bill [scale/speed?] -- I can somewhat understand the not relying on SQS but then there is anyways a reliance on EC2 right ?

[bjt]

I thought the same thing about Rabbit. PinLater's ACK system feels like a questionable reinvention of AMQP. I'm probably missing something.

Rabbit's advertised throughput doesn't reach 100,000/sec, but you should be able to shard your way around that. Or use Apache Kafka, which has the sharding built in.

[fintler]

Just for comparison, LinkedIn is doing about 2.5 million messages per second (on average per day) with Apache Kafka.

http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-...

[hatred]

Precisely, this is the same query I had when I glanced down the post. It will be great if the author or someone from the team respond to the various trade-offs and why they did not go ahead with Kafka ?

[omrvp]

(I am the author of the blog post). I posted a reply below earlier this morning that hopefully addresses your question.

[bigtones]

Looks like they needed to have built-in support for time based scheduled execution of jobs at a specific time in the future. Rabbit MQ does not have that (that I'm aware of) without doing some horrible hacks around one queue per message.

[kondro]

ActiveMQ's had that for a couple of versions though. Another highly performant/available, atomic, consistent, durable and mature queueing server.

[cyounkins]

Correct. Celery's implementation dequeues but doesn't acknowledge the task, holding it until the timer is reached.

[zackkitzmiller]

I'm curious here as well.