Hacker News new | ask | show | jobs
by stvltvs 729 days ago
So you have never needed to write SQL in your ETLs? I guess it depends on whether you consider that code, but at least 50% of the time I have to use SQL for one reason or another rather than just pointing the tool at a source and target.
1 comments

Honestly? If you are making the transformation in SQL you've lost the T to the database server. Does it mean we don't do it? Is water wet?

The reality is that we shouldn't be making the transformation in written data, now you have staging tables, procs, views, resulting tables procs, views, a staging database (often called staging or etl), and an unhappy DBA who yells at you every time you cross-apply incorrectly.

ETL should be done before the data lands in SQL.

I think you're describing ELT, an approach that I agree with and try to implement as often as possible.

I think that's orthogonal to the occasional need to use SQL to extract source data. And it's hard to beat the performance of RDBMS for your transformations. It's almost their whole raison d'ĂȘtre.