Ask HN: What Are Your Favorite Tools for Data Integrity in Data Engineering?

[gvaishno]

Hello HN community! I'm working on a data engineering project, and I'm keen to hear about the tools that you find most effective for ensuring data quality and integrity. Whether it's ETL processes, data validation techniques, monitoring solutions, or any other aspect of data engineering, I'd love to learn from your experiences. What is your go-to tools and best practices for maintaining trustworthy data throughout its lifecycle? Any insights and recommendations would be greatly appreciated!

[frankacter]

> Whether it's ETL processes

For ETL/data pipelines, tools like Apache Airflow, AWS Glue, Azure Data Factory provide flexible orchestration and monitoring. They also ensure data is properly validated, cleaned, standardized at each step.

> data validation techniques

For data validation, Spark/Python libraries, Looker Data Literacy, Great Expectations are effective for formalizing validation rules and checks on type, format, range, uniqueness etc.

Tools like Databricks Profiling, Alteryx Profiler help understand data structure, anomalies, quality issues before modeling or analysis.

For MDM/lineage, master data hubs like Talend MDM combined with tools like Apache Atlas/Collibra provide 360-degree view of data assets.

>monitoring solutions

Tools like DataDog, Prometheus, Interana are useful to monitoring data quality metrics and exceptions.

For us, the key is taking a holistic approach - validate your data at source, during transformation and at destination. Automate as many checks as possible and monitor quality continuously to ensure data reliability across its lifecycle.

[gvaishno]

Any tools specifically to ensure data integrity when data is transferred between two points A to B.

[frankacter]

That depends a lot on your environment, but I can generalize a few scenarios that are more common.

Apache Kafka for example, is an open source open-source distributed event streaming platform that, among other things, provides mechanism for data integration to ensure end-to-end data transfer.

If it is log data, Apache Flume aggregates and moves large amounts of log data efficiently. Ensures data is not lost during transfer.

Apache Spark Structured Streaming, for stream processing, it provides exactly-once semantics to guarantee data is not lost or duplicated during transfer.

Apache NiFi is another open source ETL tool that allows transferring data between systems reliably while ensuring integrity through versioning, provenance etc.

Python libraries like Fleep, Tenacity help make data transfers fault tolerant and ensure retries/rollback on failures. Integrity can be checked through hashes.

Node.js libraries, streams like StreamData allow building fault tolerant data pipelines while ensuring integrity through FlowFile handling.

Azure Data Factory provides reliable data transfer mechanisms like replication, retries, monitoring to guarantee end-to-end transfer without data loss.

[iknownothow]

You could approach data quality testing as if you're testing another piece of software by writing tests. We use dbt and it makes writing tests against models (think tables in a db) very easy.

For example, if you have a regional_orders table. You write tests in SQL to test your assumptions about that data:

* I expect regional_orders table to contain no duplicates entries.

* I expect regional_orders to ship to only a specific region.

* So on...

This has worked fairly well so far for me. But are these kinds of tests sufficient? Am I missing something?

[leandot]

It really depends on your type of data, but from the top of my mind - Pydantic.