Hacker News new | ask | show | jobs
by hulahoof 1588 days ago
I don't think you are missing anything, but allowing DBT to contain all the models that make up your various pipelines and reference each other mean that you can schedule your various pipelines at different cadences and use tags to refresh the relevant DBT models from a single code base.

It sounds like in your approach this would be writing this dependency logic into each DAG you schedule on airflow.

In the same way you would interpolate your jinja SQL before copying it into the database, you would use dbt compile or the output from a dbt run from the target/ folder and copy that SQL into your DB console or to share.

EDIT: This means your T is a single airflow node in each DAG, though I then still use airflow for the E/L tasks around it