|
|
|
|
|
|
by gurubavan
1679 days ago
|
|
|
That sounds like a great plan. We're planning to build our public API and CI/CD integrations early next year, so that developers can know what the downstream impact of their changes might be, and whether it could introduce unexpected results. We may be able to slot right in there with Pact. Mitigating the impact with monitoring is where we're at right now, but we're with you that preventing errors can be even more important. If it's interesting to you, we're happy to open up a shared slack channel to dig into the nuance as well! Just email me (guru@metaplane.dev) with the email you'd like to be added. |
|
|
When Nick Schrock created dagster, he argued that many "data cleaning" tasks which people attribute to "data engineering" aren't actually "cleaning", but are architecture problems. I believe schema changes also fall into this category. I'm extremely new to data engineering, but when I think about "What are the things which will break this system?" an application engineer thinking "I'm going to rename this column and my tests pass, so this should be fine" will break things all the time. (Similar goes for dropping a column, changing a one-to-many into a many-to-many)