Hacker News new | ask | show | jobs
by oavdeev 3487 days ago
We've looked at Airflow. Two main things why we went with Luigi are

* time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron.

* We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.