Hacker News new | ask | show | jobs
by JBerlinsky 5398 days ago
RabbitMQ (or any AMQP broker) sounds like it would do the job, since it sounds like you need to send very little data (a URL) to the background worker. I believe that there are some good AMQP/Rabbit gems for Ruby, but I'm a little dated on that. The basic premise is that a broker (RabbitMQ-Server, etc.) holds onto a queue of messages, that are distributed to N subscribers when they ask for them. The more subscribers you have to the broker, the more jobs can happen at the same time.

However, RabbitMQ might be too much overhead for what you want to do. With Ruby, specifically, I've had a good deal of experience with Resque, which uses Redis (key-value store) as a queueing system, much like RabbitMQ. It's easy to set up, and gets the job done just as well.