Hacker News new | ask | show | jobs
by aelzeiny 1690 days ago
All good choices, but it's also worth noting that the structure of the author's DAG is strange. The way this is set up, you would have to change Python code to run (or rerun) this task for different days.

In canonical Airflow, the job would be one DAG, and each day would be a separate DAG run. Then you would backfill all the days that you would like the job to run. If there's some sort of max-concurrency requirement, that would be handled by setting the `max_active_runs` parameter or by using Airflow's pool concept.

If I had to venture a guess, the author is not an experienced Airflow user, and just wanted to give a new technology an honest try.