Hacker News new | ask | show | jobs
by rhombocombus 2849 days ago
How does it compare to big iron enterprise ETL tools like IBM datastage? I have only dabbled but it looks far more appealing for a variety of reasons.
2 comments

For context, I used Datastage, Informatica, Ab Initio and SSIS in previous lives an went on to write the first version of Airflow. I developed a taste for pipelines-as-code while working at Facebook using an internal tool that is not open source.

I'd argue that pipelines as code, as opposed to dragndrop GUIs, is a better approach, at least for people who are comfortable writing code. Code is easy to version, test, diff, collaborate and allows for the creation of arbitrary abstractions.

ETL tools just can't compete with a tool that forces code to do anything. It might seem backwards, but we've abandoned all non-code environments and force pure-code for configuration for all of our pipelines.