[tedmiston]

I run an Airflow instance that does millions of tasks per month across dozens of DAGs. There's some performance tuning involved in the configuration file and of course you need the underlying resources available but Airflow has scaled to this level well for us.

If you are able to reproduce and can post to the dev mailing list, we are happy to help... especially so if it gets you off of a proprietary tool written in Perl ;).

1.10 was just released and adds a ton of commits. I'd really encourage you to give Airflow another shot if you have the time.

[vgy7ujm]

Millions per MONTH is just ridiculous. Stay with what works, revisiting this slow Python tool will be a cost and time sink. A mistake companies let engineers with millennial complex do all too often.