[catch23]

i think mongodb fills a need that isn't covered by redis or sql -- people still need semi-relational data that can scale beyond 100M rows.

redis is a nice memcached or scalable data structures replacement. we use it as a simple rabbitmq replacement.

[henryprecheur]

Yes. I didn't want to criticize MongoDB or SQL based databases. Just point out that Redis' protocol is small and easy to implement.

[riffraff]

why did you have the need to replace rabbitmq at all? (I love redis, but I'd expect using something designed specifically for X is better than using something else that also kind of does X)

[moe]

RabbitMQ is a complex beast and was, last time I used it at scale, ridden with problems under load. We had extremely serious issues like spontaneous lockups.

We eventually abandoned it when we realized our queueing needs could be modeled in Redis. In a way that we fully control, understand and can debug. And it wasn't even much work!

I'd argue this is highly preferable unless your project really needs complex routing of the kind that only AMQP can provide. Most projects don't.

[StavrosK]

I use redis to replace RabbitMQ as well, but it's all abstracted away in Celery. A question: Don't you need polling when you use redis in this way? Does it have some sort of notification functionality, apart from the recently added pubsub?

[moe]

Our queues are implemented using BLPOP which blocks until items arrive, so no polling is needed.

The newer pub/sub stuff promises to be very useful, too, but we didn't have a use-case for that, yet. Our app generally needs messages to persist until they are consumed.

What I can say however is that none of the concerns we initially had about performance/scalability held any water.

We are still running on a single redis instance (plus a slave) on a moderately sized server and it happily processes 100 messages/sec average between 4 producers and a varying number of consumers (20-100).

To add insult to injury our monitoring metrics show that this isn't even a worthwhile load for redis. The server is nowhere near breaking a sweat, the CPUs barely drop below 90% idle, there's no disk i/o to speak of, and the memory usage is more than reasonable (plenty headroom for our purposes).

Thus "just throw it at redis" has long become a common stopgap meme in this particular project. And so far we didn't have to replace any of these supposed stopgaps with something else.

[StavrosK]

Ah, very nice. It seems that I shouldn't have any qualms about throwing redis at most of my problems then, thanks!

[catch23]

in case you're wondering, we use a simple queueing thing called redpack:

https://github.com/luxdelux/redpack

[JulianMorrison]

http://code.google.com/p/redis/wiki/BrpoplpushCommand

[riffraff]

thanks for the reply, I can see if you make no use of functionalities other than push/pop how redis would be just fine.

Could you be so kind to qualify what load are we talking about, as producers/consumers & message rate/size/persistence level?

I have used ActiveMQ with systems of about ten nodes and a message rate of <2k/sec and it worked fine, and I always believed rabbitmq was faster/more stable (erlang bias!) so if your volume is 500k/s I'm ok, otherwise I'd file this as another slightly worrying information about rabbitmq (after the problems I heard from reddit)