| "I want to write my orchestration in Python and I'm comfortable hosting my own compute" -> Prefect (lightweight) or Dagster (heavier but featureful) "My team already knows Airflow and/or I want to pay Astronomer a lot of money" -> Airflow "I love YAML and everything is on k8s anyway" -> Argo "I just want something that works out of the box and don't want to host my own compute" -> Shipyard, maybe Orchest "I want a more flexible, generic workflow engine and don't care about writing orchestration in Python" -> Temporal/Cadence "I am very nostalgic" -> Azkaban, Oozie, Luigi "I love clunky Java solutions to data problems" -> Nifi et al "I like to pay for half-managed solutions and late upgrades to a first-generation technology" -> AWS/GCP hosted Airflow options "I am on AWS and it doesn't need to be complicated" -> AWS Step Functions |
I know there's a degree of oversimplification going on here, but there's something to be said for having a simple bullet-list breakdown of all the use-cases - alongside the best tool for each use-case.
It is servers as a practical starting point in terms of narrowing down the list of tools (of which there are so many), before one proceeds with a deeper dive into the best fitting tool.
Would be great if there were a site that did this sort of thing for all the common architectural needs.