Hacker News new | ask | show | jobs
by RocketSyntax 1829 days ago
Or Apache Airflow. I'd expect this and KFP to have better integrations that cater to data science. You can use Elyra extension to design the workflows right from JupyterLab.
1 comments

Not from said company, but did do a workflow tool comparison at my work and we also went with Argo.

Argo had better kubernetes + surrounding ecosystem integration out of the box, it was designed to run containers by default which suited us because we had mixed language workloads. Airflow was mostly Python specific, unless you then ran plugins and extensions, the config/pipeline definition was written in Python which I didn’t want to do after witnessing my teammates write the worst Python I’ve seen in my career, and last time I evaluated it, it depended on a bunch of external, Python specific tools (celery etc) that I had previously found painful to run.

> my teammates write the worst Python I’ve seen in my career

Isn't that fairly common which is why there are "ML engineers" that then productionize (clean up and optimize) the original code to be plugged into a production workflow / pipeline system?