|
|
|
|
|
by caravel
3397 days ago
|
|
At a high level [for Airflow specifically] the scheduler or workflow engine cares most about the tasks and their dependencies, and is somewhat agnostic about the units of work it triggers. It's possible to use a feature called XCom as a message bus between tasks, but would typically direct people in the direction of having stateless, idempotent, "contained" units of work and avoid cross task communication as much as possible.
https://airflow.incubator.apache.org/concepts.html#xcoms For your case [which I have little input on] I think singleton DAGs described in another post on this page may work. |
|