[peakaboo]

It's always interesting how people's experiences are so different. At many of my jobs, airflow has just worked perfectly and we never even had to think about it.

But i welcome alternatives since I think it has become way too complicated now, and it's not necessary for most people with all the complexity (distributed tasks with celery comes to mind).

[uskrokette]

Yes I agree with you. Airflow largely does the job in many cases, but when you reach its limits, which is clearly the case quickly, you are really blocked and it can become the hell and a considerable waste of time and efficiency.

For my opinion, Airflow has bad design on files: Airflow does not understand the files it processes so if exporting data, we have to adapt each source with each destination.

Scalability is not thought out and it's therefore very complicated: Airflow was designed as a single node and not scalable, so when it extracts data, it cannot easily pass the file to a next task in a distributed environment, which generally requires having to redevelop a task that includes the from > to Airflow uses a lot of cpu on its own, it takes a huge architecture to manage a lot of flows...

in my case, I work on big projects and I need scalability which prefect offers me and now this new Kestra data orchestrator.

say me what do you think about Airflow'friends!