[rektide]

> Airflow and Dagster, along with similar tools like Flyte, Luigi, Argo Workflows, and Prefect, are bucketed under a set of systems you could utilize for the orchestration of jobs.

Quite the list of tools, nice. Interesting space to me.

[davidmirror]

I'm not sure we can label one solution to be "better than" the other just because it "is leveraging containerization to entirely solve..." anything. Reasons:

1. Containerization addresses the monolith pattern issues yes, but introduces other challenges (data persistency, image management, immutable infra, etc) 2. It takes much more than just containers and executors to design a reliable platform for ML/Data.

I think what can set projects apart is how/if they make use of Software Engineering patterns for what matters to ML as a workload. An orchestrator that could leverage a control loop (reconciler) pattern, similar to what K8s does for containers, would provide the eventual consistency and reliability required by most ML workloads.