[uskrokette]

Yes I agree with you. Airflow largely does the job in many cases, but when you reach its limits, which is clearly the case quickly, you are really blocked and it can become the hell and a considerable waste of time and efficiency.

For my opinion, Airflow has bad design on files: Airflow does not understand the files it processes so if exporting data, we have to adapt each source with each destination.

Scalability is not thought out and it's therefore very complicated: Airflow was designed as a single node and not scalable, so when it extracts data, it cannot easily pass the file to a next task in a distributed environment, which generally requires having to redevelop a task that includes the from > to Airflow uses a lot of cpu on its own, it takes a huge architecture to manage a lot of flows...

in my case, I work on big projects and I need scalability which prefect offers me and now this new Kestra data orchestrator.

say me what do you think about Airflow'friends!