|
|
|
|
|
by Hippocrates
1416 days ago
|
|
I have not used celery + rabbitmq but I assume that combo is like sidekiq + redis, or any other job queue + worker system. Airflow packages those things together and adds some additional features
- UI with Graph, gantt, logs and other views of the workflow
- Users and permissions
- Places to store config
- Mechanisms for passing small data between tasks
- Various "sensors" for triggering workflows
- Various operators that interact with common data-oriented systems (bigquery, snowflake, s3, you name it). These are basically libraries that expose a config-forward API. Probably the main selling point is the pre-made operators, but in short it is a complete solution with bells and whistles that aligns itself with the data ecosystem. |
|