[glogla]

Yeah, Airflow isn't multi-tenant.

People can potentially overwrite each other's DAGs. Credential management is complicated. Broken DAG can stop whole Airflow. Slow DAG can impact performance of whole Airflow. Getting DAGs to wait for each other (like one team prepares data up to a point and then other team builds on that) is kind of a nightmare. Sometimes people want features from newer Airflow, but some other team built DAG that isn't forward compatible. Etc etc.

But I'm not sure there actually is a better solution elsewhere. At least I have not seen it yet, maybe Dagster is on a good road.

But as I said, for centralized solutions it works really well.

[kumare3]

Some of these were the core problems that we wanted to address as part of https://flyte.org. We started with a team first and multi-tenant approach at the core. For example, each team can have separate IAM roles, secrets are restricted to teams, tasks and workflows are shareable across teams, without making libraries. and it is possible to trigger workflows across teams. Each teams workflows are tasks and grouped using a construct called projects. It is even possible to separate execution clusters per team, per workflow onto separate k8s clusters. Also the platform is built to be managed and easily deployed.