Hacker News new | ask | show | jobs
by chakspak 1957 days ago
My company has deployed Kubeflow for production model training. Early on, we used their big deployment, but we got frustrated trying to manage it with kfctl, so we started using kustomize directly and deploying only what we need, like KFP. So no istio for us! YMMV with your use case.
1 comments
FridgeSeal 1957 days ago
Oh this is perfect to hear!

Yeah my plan is to use it the same way we use kubernetes: everything via kubernetes configs/justo use is friends and nobody using CLI’s, which I’m convinced are the text version of “click ops” hahaha.

So Istio isn’t strictly required? Can I ask which components you deploy? At minimum I’m planning on just the deployment and serving components and just use straight Polyaxon for training.

link
chakspak 1956 days ago
Yeah, there are quite a few applications that actively discourage the use of anything other than their CLI, which I find profoundly bizarre. Istio had a big warning in their docs that their Helm charts were deprecated, even though istioctl uses them under the hood. Funny that Kubeflow has its own Istio deployment, and they make you use kfctl to deploy it, haha.

It's not strictly required, but might be for you, since A/B deployment actually is a thing Istio does. We only use Kubeflow Pipelines currently; looking into Katib.

My feeling about Kubeflow is that it's a package of a lot of things that exist, with nice things on top, and an easy way to deploy those things. Only, it ends up not being that easy, and not easy at all to configure, and various features of the underlying tools are hard to get to or completely unavailable.

If you deployed your own Istio, you'd understand it end-to-end and could solve problems with it when it goes south, and you'd even understand what exactly you're using it for and why. The biggest problem I have with kfctl is that it basically asks for system:masters privilege to install everything and then you get to figure out what it did on your own. I don't want Istio, Cert-Manager, Knative, Argo, etc. deployments that I don't understand and can't easily configure because they're buried 5 levels deep in Kustomize overlays. These are all things I can install from elsewhere and with more documentation.

Kubeflow Pipelines is still a little mysterious, but the footprint is much smaller and not as invasive.

Never used Polyaxon. Gonna be Googling that!

link