[thundergolfer]

Having to setup and run Airflow on K8s is a hell of a prerequisite step to getting cost-efficient and fast access to GPU training.

Airflow is also absolutely not built for that purpose. It's ~10yr old Hadoop-era technology.

[__MatrixMan__]

As for getting airflow on k8s in the first place, the apache airflow helm chart pretty much just handles things, doesn't it? It might be a pain to manage many deployments for many teams, but going from 0 to 1 isn't so bad.

As for configuring the kubernetes pod operator to ask for pods with GPU's, it exposes the k8s python API in the dag definition. I haven't done it myself, but I think that it's not really airflow that's going to be a pain there. Getting the pod spec right is gonna have to happen whatever does the orchestration.

(Full disclosure: my employer offers airflow as a service)