|
|
|
|
|
|
by boulos
2853 days ago
|
|
|
Disclosure: I work on Google Cloud. We ended up choosing Airflow for our managed workflow service, Composer [1]. One of the main cons is that this space is generally fragmented. When you’re starting out with a simple repeatable task, the natural default is just a cron job (and if you’re “fancy”, using Jenkins or similar to kick it off). By comparison, getting started with Airflow means setting it all up, and then writing a Python script that represents your DAG. It may be better hygiene, but that’s not people’s first preference. Beyond the obvious “this thing is a real workflow orchestration system, it handles failures and dependencies”, I think the main advantage is the pre-built Operators. Instead of being just given a blank bash or python script, this is a community-driven effort to avoid everyone needing to roll their own. It’s still a young community, but growing quickly (and we’re intent on pushing). [1] https://cloud.google.com/composer/ |
|
|
Our experience was similar to boulous' — Airflow is awesome once it's running but getting it running in an environment that scales to the point that you can deploy your first production DAG can take some effort. That's what led us to trying to do that work once for everyone.
Reusability and composability of components are some of my favorite aspects of working with Airflow.
[1]: https://www.astronomer.io/
[2]: https://github.com/astronomerio/helm.astronomer.io