Hacker News new | ask | show | jobs
by shmoogy 1416 days ago
I've been using airflow for about 2 years now in production. It's been mostly good - the few times things go wrong, it's a huge pain in the ass to figure out why... but it's significantly better than just straight cron on Linux. Airflow 2 has improved a lot of speed and catching up issues from airflow 1.x

I don't have time to investigate other solutions like dagster and prefect and migrate jobs to it for testing.
1 comments
glogla 1416 days ago
If it is just you, you are fine, and I'm not sure other tools would have that much benefit.

Trouble with Airflow starts when multiple teams and user types start to share it.

link
shmoogy 1416 days ago
I've definitely noticed more issues after adding users, but it's more that they don't actually understand a lot of what they're trying to do and cause problems when writing dags.
link
glogla 1415 days ago
Yeah, Airflow isn't multi-tenant.

People can potentially overwrite each other's DAGs. Credential management is complicated. Broken DAG can stop whole Airflow. Slow DAG can impact performance of whole Airflow. Getting DAGs to wait for each other (like one team prepares data up to a point and then other team builds on that) is kind of a nightmare. Sometimes people want features from newer Airflow, but some other team built DAG that isn't forward compatible. Etc etc.

But I'm not sure there actually is a better solution elsewhere. At least I have not seen it yet, maybe Dagster is on a good road.

But as I said, for centralized solutions it works really well.

link
kumare3 1415 days ago
Some of these were the core problems that we wanted to address as part of https://flyte.org. We started with a team first and multi-tenant approach at the core. For example, each team can have separate IAM roles, secrets are restricted to teams, tasks and workflows are shareable across teams, without making libraries. and it is possible to trigger workflows across teams. Each teams workflows are tasks and grouped using a construct called projects. It is even possible to separate execution clusters per team, per workflow onto separate k8s clusters. Also the platform is built to be managed and easily deployed.
link