[hichkaker]

Thanks! We are going to add an example to our website and perhaps publish it as a Github App.

The flow is: For every whitelisted (say, SQL) file changed by a pull request, our bot compares the production data (produced by the master branch) vs. the data output of the new branch and pastes the diff as a comment to the PR. Some YAML configuration is needed to, for example, adjust it to your repo file structure.

[ishcheklein]

Thanks! And how and where does setup happens which database to use to run the query for the specific SQL file? Also if it's part of some pipeline will it have to run the whole pipeline from the very beginning?

[hichkaker]

So there are two main workflows:

1) Run diff as part of CI (on code change), that can be done via Github workflow (or other CI). The Github workflow takes in a YAML config file, and also is customizable so you can fit it according your needs. For example, one of our customers names SQL files after tables they populate, so when the bot sees a changed SQL file, it knows what table to compare it to.

2) If you are looking to run a diff within the data pipeline itself (e.g. in Airflow), you can wrap Datafold API in an Airflow operator to integrate it as a task in your DAG.

I hope that I covered your questions but please let me know if you are interested in specific use cases!