[bashfulpup]

This space is honestly a mess. I did an in depth survey around 1.5 yrs ago and my eventual conclusion was just to build with airflow.

You either get simplicity with the caveate that your systems need to perfectly align.

Or you get complexity but will work with basically anything (airflow).

[febed]

Would be interested to know what drawbacks you found with Dagster or Prefect.

[jt_b]

Prefect is amazing. Built out an ETL pipeline system with it at last job and would love to get it incorporated in the current one, but unfortunately have a lot of legacy stuff in Airflow. Being able to debug stuff locally was amazing and super clean integration with K8S.

[bashfulpup]

Other guy said it right. These work and are fine but you lose the legacy stuff. If you know your limits and where the eventual system will end up it's great and probably better.

If you are building a expandable long term system and you want all the goodies baked in choose airflow.

Pretty much the same as any architecture choice. Ugly/hard often means control and features, pretty/easy means less of both.

On the surface the differences are not very noticable other than the learning curve of getting started.

[nikolayasdf123]

+1 to this. other solutions over-promise, under-deliver, poor developer relations and communication, "open-source, but pay us" style open-source, and is indeed a mess