Hacker News new | ask | show | jobs
by davepeck 5058 days ago
Does anyone know if this has explicit support for a task queue like Celery?
1 comments
dudus 5058 days ago
I never tried it but I guess you can use SQS for queues on AWS.

http://aws.amazon.com/sqs/

link
bialecki 5058 days ago
If you go with SQS, keep in mind it doesn't guarantee that messages will be received exactly once[1]. I've used SQS for large projects and this was killer because we had to implement message locking ourselves to prevent messages from being processed twice. Since then, this has been a deal breaker for using SQS again (now use Celery).

[1] http://aws.amazon.com/sqs/faqs/#How_many_times_will_I_receiv...

link
michaelt 5058 days ago
I was under the impression [1] that a message would only be received more than once if the first attempt at processing failed to complete in the lock time - for example if the queue consumer crashed. Is it more complicated than that?

[1] http://aws.amazon.com/sqs/#functionality

link
dkersten 5058 days ago
Speaking of celery - I was looking at celery over the weekend. Do you know if its possible to submit jobs from languages other than Python (specifically, I would like to use Java)? I have some Clojure code running on Storm from which I would like to submit tasks to be executed in Python on other machines. Do you know if this is possible with celery? I couldn't really find any information on it.
link
ozataman 5058 days ago
Sorry, but lol. You had to implement locking on top of a job queueing system to make it work? That's the whole point of a queueing system!!!

You probably didn't realize you had to extend the lock peeiodically.

link