[FridgeSeal]

Oh this is perfect to hear!

Yeah my plan is to use it the same way we use kubernetes: everything via kubernetes configs/justo use is friends and nobody using CLI’s, which I’m convinced are the text version of “click ops” hahaha.

So Istio isn’t strictly required? Can I ask which components you deploy? At minimum I’m planning on just the deployment and serving components and just use straight Polyaxon for training.

[chakspak]

Yeah, there are quite a few applications that actively discourage the use of anything other than their CLI, which I find profoundly bizarre. Istio had a big warning in their docs that their Helm charts were deprecated, even though istioctl uses them under the hood. Funny that Kubeflow has its own Istio deployment, and they make you use kfctl to deploy it, haha.

It's not strictly required, but might be for you, since A/B deployment actually is a thing Istio does. We only use Kubeflow Pipelines currently; looking into Katib.

My feeling about Kubeflow is that it's a package of a lot of things that exist, with nice things on top, and an easy way to deploy those things. Only, it ends up not being that easy, and not easy at all to configure, and various features of the underlying tools are hard to get to or completely unavailable.

If you deployed your own Istio, you'd understand it end-to-end and could solve problems with it when it goes south, and you'd even understand what exactly you're using it for and why. The biggest problem I have with kfctl is that it basically asks for system:masters privilege to install everything and then you get to figure out what it did on your own. I don't want Istio, Cert-Manager, Knative, Argo, etc. deployments that I don't understand and can't easily configure because they're buried 5 levels deep in Kustomize overlays. These are all things I can install from elsewhere and with more documentation.

Kubeflow Pipelines is still a little mysterious, but the footprint is much smaller and not as invasive.

Never used Polyaxon. Gonna be Googling that!