[numbsafari]

Personally much prefer kustomize for the “ship an app” business.

Probably even better is to ship a controller and a CRD for the config.

Doing it that means you ship a schema for the parameters of the config, and that you have code that can handle complexities of upgrades/migrations that tools like kustomize and helm struggle or fail at altogether.

[cortesoft]

We switched from kustomize to helm and I really can't understand why anyone would prefer kustomize. Having the weird syntax for replacing things, having to look at a bunch of different files to see what is going on...

I love how in Helm I can just look at the templates and figure out what values I need to change to get what I want, and I love each environment only needing a single values file to see all the customizations for it.

People complain about it being a template language, but that is exactly what you need!

[fivre]

JSON patches aren't the most intuitive and kustomize needs some helper tooling to generate them for you (given JSON objects A and B, generate a patch that transforms A into B), but overall the kustomize model makes more sense, and the team behind it seems to be more actively improving developer QoL stuff than Helm is

templates are only good if your templates can remain simple and do not need to expose most of the output fields. my experience developing a chart for wide distribution has been very much that your templates will not remain simple (and will turn into an incomprehensible mess, since you'll need them to handle tasks templates are fundamentally poorly suited for) and that there is always someone, somewhere, that needs some particular resource field exposed in values.yaml. the

> As a result, the number of possibilities for configuration is often unreasonably large and complicated, mimicking the actual resources they want to create, but without any schema validation!

bit from the op is incredibly true. values.yaml grows, over time, to have every field in the objects it generates, just organized differently, without validation, and with extra complicated relationships with other settings

kustomize allowing you to provide a base set of resources that users can apply their own patches to avoids that config surface bloat problem entirely

[Hamuko]

>Having the weird syntax for replacing things

Isn't the "weird syntax" just either Yaml files or just JSON Patches, which is a pretty easy standard?

>having to look at a bunch of different files to see what is going on

I consider that a feature, not a bug. prod/larger-memory-request.yaml makes it much easier for me to see what goes into deploying the prod environment instead of for example the test environment.

[cortesoft]

By "weird syntax" I mean stuff like "patchesJson6902" or "configMapGenerator" or "patchesStrategicMerge" where you have to know what each field means and how they work.

A template is much easier to read. I had zero experience with go templating, but was able to figure out what it all meant just by looking at the templates... they still looked like kubernetes resources

As for looking at a bunch of different files, if you like having a "larger-memory-request" file, you can still do that with helm... you can use as many values files as you want, just include them in precedence order. You can have your "larger-memory-request" values file.

[miiiiiike]

That just using Kustomize. There’s a difference between learning curve and frustration post learning curve. Kustomize isn’t that bad, Helm comes with far more headaches, especially if you need to do any kind of inheritance.

Keep your customizations flat and compile them to yaml+grep to find out what’s getting overridden and where.

[ahoka]

Do you have any really complex Helm template? It either becomes an unreadable mess or you end up with basically manifest snippets in your values file.

[ljm]

You shouldn't need to read the source to configure the values how you want.

Helm's problem is that `values.yml` is basically the API and every helm chart provides its own (often incomplete) interface with poorly documented defaults. Some of those can spread over 3k+ lines and it's utterly overwhelming to figure out what to do with that.

[candiddevmike]

There nothing better than a huge comment describing behaviors above a value: {} to ruin your day.

[cortesoft]

When I see something like that, I just search in the templates directory for where the value is used.

[morelisp]

> Probably even better is to ship a controller and a CRD for the config.

Maybe it's just us, but our operations team puts pretty hard restrictions on how we're allowed to talk to the K8s API directly. We can turn a regular Deployment around as fast as we can write it, but if we needed a controller and CRD update it'd take us like three days minimum. (Which, I even sort of understand because I see the absolute garbage code in some of the operators the other teams are asking them to deploy...)

[cassianoleal]

Generally speaking, operators and CRDs are more in the domain of your platform rather than your products. They should provide common interfaces to implement the business requirements around things like uptime, HA, healthchecking, observability, etc.

If a product team sees itself needing to deploy an operator, it's likely the platform is subpar and should be improved, or the product team is overengineering something and could do with rethinking their approach.

As in most cases, a conversation with your platform/ops/devops/sre/infra team should help clarify things.

[jen20]

If you run a multi-tenant Kubernetes cluster at scale, operators with poor discipline spamming the API servers and taking etcd down is a leading cause of sadness.

[morelisp]

This is the common view among our ops team, sure, but for a vocation so prima facie obsessed with postmortems/five-whys/root-causes/etc it's depressingly shallow.

[mountainriver]

Controller + CRD is the way to go and seems more in line with how k8s was intended to be used.

The challenge has historically been that controllers are a lot harder to write, but I think that story has improved over the years

[arccy]

operators are great when you control it. less so when it's some third party one that doesn't support that field you need on a resource it creates

and all the customizations just end up being yaml merges from a configmap string or CRD if you're lucky

[mountainriver]

Fair enough, the UX is just so much better that I'd gamble it in most use cases

[jpdb]

> Probably even better is to ship a controller and a CRD for the config.

But how do you package the controller + CRD? The two leading choices are `kubectl apply -f` on a url or Helm and as soon as you need any customization to the controller itself you end up needing a tool like helm.

[numbsafari]

Just use kustomize. It’s baked into kubectl. No need for a separate tool.

[cassianoleal]

Agree. I'd recommend to start with static YAML though. Use kustomize for the very few customisations required for, say, different environments. Keep them to a minimum - there's no reason for a controller's deployment to vary too much - they're usually deployed once per cluster.