[jimmytucson]

If I put each select statement in its own Airflow task, I get the same lineage dbt gives me, except I can see it and administer to it alongside all my other E and L-type tasks.

Also, I can write my T in plain ol’ SQL (granted, with some jinja) instead of this dbt-QL that I can’t copy and paste into my database console or share with a non-dbt user.

So, folks who have adopted dbt: what am I missing by being a fuddy-duddy?

[hulahoof]

I don't think you are missing anything, but allowing DBT to contain all the models that make up your various pipelines and reference each other mean that you can schedule your various pipelines at different cadences and use tags to refresh the relevant DBT models from a single code base.

It sounds like in your approach this would be writing this dependency logic into each DAG you schedule on airflow.

In the same way you would interpolate your jinja SQL before copying it into the database, you would use dbt compile or the output from a dbt run from the target/ folder and copy that SQL into your DB console or to share.

EDIT: This means your T is a single airflow node in each DAG, though I then still use airflow for the E/L tasks around it

[blakeburch]

Biggest factors in dbt adoption are:

- Automatic DAG generation based on dbt-QL declared dependencies.

- The structure of where (db/schema) and how (table/view/temporary) things are built is defined in a YAML configuration, not the individual SQL statements.

- Testing/documentation baked in.

Sure, you can manage every select statement as its own task, but it becomes pretty infeasible once things scale.

dbt can still be administered alongside all other E and L-type tasks. It's just a Python CLI wrapped around SQL SELECT statements.

[burkaygur]

Looks like dbt Labs is working on a dbt Server that can translate your dbt-QL type queries into SQL. That way you can use different "building blocks" of SQL with the rest of your team. This becomes pretty powerful with the metrics layer that dbt just introduced.