[aidos]

I have long running jobs (say, 5 minutes on average - up to an hour). I originally used celery (after picloud shut down) but it just doesn't work well with those charcteristics. Each worker reserved an extra job so it was impossible to get good cpu utilisation.

I switched to rq and it's all been much easier. The behaviour is easy to understand and it's easy to inspect redis to see what's going on.

In terms of the code restart angle - I'm fairly sure you can effectively restart the workers. They run as a single process that forks to do work. Each copy you run only has a single worker, so you need to run multiple instances yourself. If you kill the parent it waits until the child has finished the job it is on before terminating.

I could be wrong about some of the details. I'd recommend giving it a shot. I must have run 100,000s of jobs through it now and I haven't had a single issue.

http://python-rq.org/docs/workers/

[ThePhysicist]

Thanks, that sounds interesting!