[js8]

Can't agree more.

IMHO, if you want to make a dent in the space, figure out better debugging tools!

In particular, tools that explain how a certain (specific) value was calculated in the system, tools that let you bisect the source data in some way and let you focus on the source data that are likely to have a problem, tools that help you figure out that certain intermediate value in calculations is an outlier, tools that let you test certain assumptions about data over the whole pipeline..

[veritas3241]

You're talking about more debugging tools within the transformation steps of a pipeline, right? dbt is helping with that via data tests (see https://gitlab.com/meltano/analytics/tree/master/elt/dbt/tes...) as an example.

I'd love for a more robust way to test data pipelines and the data within them generally. I was at DataEngConf earlier this year and many people were talking about this problem exactly. One way we're trying to address it a bit is by using the Review Apps feature on Merge Requests within GitLab. Right now, when you open an MR on our repo it will create a clone of the data warehouse that's completely isolated from production. This, obviously, can't scale once the DW is beyond a certain size, but I think there are ways to keep this sort of practice going.