[edumucelli]

More recent tools such as Dagster and Prefect have much more to offer. One simple example is communication between tasks. Airflow has a clunky system for that called XCom. The actual author of XCom says you probably should not use it due to the level of hackery it has under the hood [1]:

On Dagster and Prefect you communicate between tasks as if you were writing pure Python. On Airflow on the other hand ...

[1] https://www.youtube.com/watch?v=TlawR_gi8-Y&t=740s

[marcinzm]

I'm not sure if linking to a talk from 2018 in 2022 for a project that is being actively worked on (a bunch of Python abstractions for xcom were added in 2.0) is fair.

[byteflip]

Yes, curious if the Taskflow API introduced as part of Airflow 2.0 reduces this pain. It appears much easier/saner than working with XCOMs directly - less coupling and removes the need for lots of boilerplate code.

[karog]

There's also Flyte, which is natively data aware and schedules tasks around data dependencies. The syntax is essentially pure python too.