[nathanscully]

We are using Airflow to manage ETL jobs. Nearly all of these are SQL steps dynamically generated via an Airflow DAG that transform transaction and event data on our SQL warehouse into 'master' tables everyone has access to. All SQL and DAG code is committed into Github and we have a process to update Airflow and merge any changes after its peer reviewed. Every change is done via a PR so we have visibility and accountability.

One thing we want to improve is our testing component, curious to hear how people manage test workflows, replicating prod before promoting new pipelines. I.e. I want the branch to run a full test suite against a prod replica before automatically replacing the current prod pipeline.