| I had participated in migrating around 100 fairly complicated pipelines from Airflow to Dagster over six months in 2021. We used k8s launcher, so this feedback does not apply to other launchers e.g. Celery. Key takeaways roughly those: - Dagster's integration with k8s really shines as compared to Airflow, it is also based on extendable Python code so it is easy to add custom features to the k8s launcher if needed. - It is super easy to scale UI/server component horizontally, and since DAGs were running as pods in k8s, there was no problem scaling those as well. For scheduling component it is more complicated, e.g. builtin scheduling primitives like sensors are not easily integrated with state-of-art message queue systems. We ended up writing custom scheduling component that was reading messages from Kafka and creating DAG runs via networked API. It was like 500 lines of Python including tests, and worked rock-solid. - networked API is GraphQL while Airflow is REST, both are really straightforward, however in Dagster it felt better designed, maybe due to tighter governance of Dagster's authors over the design. - DAG definition Python API, e.g. solid/pipeline, or op/graph in a newer Dagster API, is somewhat complicated as compared to Airflow's operators, however it is easy to build custom DSL on top of that. One would need custom DSL for complicated logic in Airflow as well, and in case of Dagster it felt easier to generate its primitives, than doing never ending operators combinations in case of Airflow. - Unit and integration testing are much easier in Dagster, the authors put testing as a first-class citizen, so mocks are supported everywhere, and the code tested with local runner is guaranteed to execute in the same way on k8s launcher. We never had any problems with test environment drift. The biggest caveat was full change of internal APIs in 0.13, which forced the team to execute a fairly complicated refactor, due to deprecation of the features we were depending on e.g. execution modes. Had we spent more time on Elementl slack, it would be easier to put less dependencies on those features ^__^ |