[estefan]

I've ported several reasonably complex jobs (files delivered to FTP at arbitrary times to be run through several Hadoop jobs) to luigi and it's been very good. Much more resilient than trying to use something that can only schedule jobs at specific times of the day.

It also has few dependencies and is lightweight (i.e. it's all python, so no JVM tying up resources).

[haksmak]

fwiw, it is not the case that pinball can schedule jobs only at specific times of the day. In fact the scheduler is merely a special type of worker that happens to start new workflows. It is totally doable to kick off a new workflow at any point in time, bypassing the scheduler.

Also, Pinball is also all Python but it currently has a dependency on mysql so it is definitely not as a lightweight as a standalone tool as luigi but it also offers much more in terms of the available features.