Hacker News new | ask | show | jobs
by flarco 6 days ago
It does seem over-engineered.

Normally you'd keep all the transformation in one place in the "data warehouse", which could be Postgres/Redshift/Snowflake/Clickhouse etc. Using some el tools (like sling cli for example) to load into the DW, and then using a transformation tool (like dbt for example) to do all the transformations in SQL, source controlled. It's advantageous to keep all of the transformation logic in one place (to avoid mismatches as you've mentioned). And from the DW, after data is ready, various tools/clients can consume it (such as BI tools, CRMs, APIs, AI, etc). All aligned.

1 comments

Thanks. What you described is much more what I'd expect a data warehouse process to look like. Which is driving me mad because I don't understand why there are so many steps with so many tools.