|
|
|
|
|
|
by mmq
2135 days ago
|
|
|
Argo implements its own scheduler AFAIK, otherwise how would it manage dependencies and the execution graph. The part that Argo is using K8S for is orchestration, which Airflow can do as well with the KubernetesPodOperator, but it's not a cloud native solution and it spins the whole scheduler and backend for each task. We tried to make Polyaxon[0] work with Airflow for Machine Learning specific workflows, but it was very painful and it does not have a good state/artifacts management, which leaves the users tweaking around. We end up making a simple abstraction on top K8S, much easier, to provide features for parallel executions, dependencies, failure handling, retries, ... as well as handling ML specific graphs such as hyperparameter tuning and distributed scheduling. [0]: https://github.com/polyaxon/polyaxon |
|
|
By the way, Polyaxon looks awesome, I’ve been wanting to try it for a while, but just don’t have any machine learning projects in the pipeline at the moment alas.