Hacker News new | ask | show | jobs
by meritt 2913 days ago
Tangential to the point of the article but when did writing a select query become an "ETL process"?
1 comments

Suppose you have a data source and business logic which you want to run periodically on the data source. Here are two scenarios which you could reasonably implement this as:

Method one: You write a SQL query and some Python. You put a sticky note on your computer "Remember to run that biweekly."

Method two: You pull up your shop's documentation for how to add the (BIG_NUMBER)th entry into the data processing pipeline. This gets you automatic scheduling, retries, monitoring, audit trails, alerts to the right people in case of breakage, etc etc. You write a SQL query and some Python. You plug it into the existing infrastructure.