|
|
|
|
|
|
by __MatrixMan__
1168 days ago
|
|
|
As for getting airflow on k8s in the first place, the apache airflow helm chart pretty much just handles things, doesn't it? It might be a pain to manage many deployments for many teams, but going from 0 to 1 isn't so bad. As for configuring the kubernetes pod operator to ask for pods with GPU's, it exposes the k8s python API in the dag definition. I haven't done it myself, but I think that it's not really airflow that's going to be a pain there. Getting the pod spec right is gonna have to happen whatever does the orchestration. (Full disclosure: my employer offers airflow as a service) |
|
|