[uskrokette]

Hi HN, Im a big fan of Airflow and its advantages on workflow management tools. After many problems of performance on a project I try some new modern platform, prefect dagster. Very good ! And recently I read this medium blog on Kestra a new platform and I decide to try it. It was a good decision because the user experience was cool, easy, efficient and powerful. The engine seems much more advanced to me because the same complex flows are much faster on Kestra than with Prefect. Take time to interest about and up this project on https://github.com/kestra-io/kestra

[peakaboo]

It's always interesting how people's experiences are so different. At many of my jobs, airflow has just worked perfectly and we never even had to think about it.

But i welcome alternatives since I think it has become way too complicated now, and it's not necessary for most people with all the complexity (distributed tasks with celery comes to mind).

[uskrokette]

Yes I agree with you. Airflow largely does the job in many cases, but when you reach its limits, which is clearly the case quickly, you are really blocked and it can become the hell and a considerable waste of time and efficiency.

For my opinion, Airflow has bad design on files: Airflow does not understand the files it processes so if exporting data, we have to adapt each source with each destination.

Scalability is not thought out and it's therefore very complicated: Airflow was designed as a single node and not scalable, so when it extracts data, it cannot easily pass the file to a next task in a distributed environment, which generally requires having to redevelop a task that includes the from > to Airflow uses a lot of cpu on its own, it takes a huge architecture to manage a lot of flows...

in my case, I work on big projects and I need scalability which prefect offers me and now this new Kestra data orchestrator.

say me what do you think about Airflow'friends!