[Terretta]

> dlt is a minimalistic Python library, requires no backend and works whenever Python works. You can finetune it to ... run with ... Dagster.

Relating to dagster in particular, this is in your docs:

dlt incorporates the concept of implicit extraction DAGs to handle the dependencies between data sources and their transformations automatically. A DAG represents a directed graph without cycles, where each node represents a data source or transformation step.

When using dlt, the tool automatically generates an extraction DAG based on the dependencies identified between the data sources and their transformations. This extraction DAG determines the optimal order for extracting the resources to ensure data consistency and integrity.

How do you think about tying and running this and dagster together?

[adrianbr]

There are multiple ways to run together - we will show a few in a demo coming out soon.

We also consider a tighter integration like with Airflow described here as a possible next step https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deplo...

We will investigate the interest incrementally as to not build any plugins that don't end up used.

[nerdponx]

For an example of prior art, you should look into Astronomer's Cosmos Library to see how they integrate Dbt into Airflow.

[adrianbr]

Thank you! That's the example we looked at for our dlt-airflow integration :) the dlt dag becomes an airflow dag.