[empath-nirvana]

I think helm is at it's best when you need to _publicly distribute_ a complex application to a large number of people in a way that's configurable through parameters.

For internal applications, it's in an awkward place of being both too complex and too simple, and in a lot of cases what you really want to do is just write your own operator for the complex cases and use kustomize for the simple cases.

Most of the problems with updating and installing helm charts go away if you manage it with something like argocd to automatically keep everything up to date.

[evancordell]

This is interesting, I have the opposite opinion. I dislike helm for public distribution, because everyone wants _their_ thing templated, so you end up making every field of your chart templated and it becomes a mess to maintain.

Internal applications don't have this problem, so you can easily keep your chart interface simple and scoped to the different ways you need to deploy your own stack.

With Kustomize, you just publish the base manifests and users can override whatever they want. Not that Kustomize doesn't have its own set of problems.

[moondev]

Kustomize also supports helm charts as a "resource" which makes it handy to do last mile modifications of values and 'non value exposed" items without touching or forking the upstream chart.

[preisschild]

There are good common libraries which expose every property by default, so you dont need to make everything template-able yourself

https://github.com/bjw-s/helm-charts/tree/main/charts/librar...

[mieubrisse]

How would you feel if you could use Starlark (if you're familiar with it) to parameterize a la Helm, and then can add more Starlark commande later to update previously-defined infra a la Kustomize?

Full disclosure: our startup is trying to build a tool where you don't have to pick, so trying to test the hypothesis

[numbsafari]

Personally much prefer kustomize for the “ship an app” business.

Probably even better is to ship a controller and a CRD for the config.

Doing it that means you ship a schema for the parameters of the config, and that you have code that can handle complexities of upgrades/migrations that tools like kustomize and helm struggle or fail at altogether.

[cortesoft]

We switched from kustomize to helm and I really can't understand why anyone would prefer kustomize. Having the weird syntax for replacing things, having to look at a bunch of different files to see what is going on...

I love how in Helm I can just look at the templates and figure out what values I need to change to get what I want, and I love each environment only needing a single values file to see all the customizations for it.

People complain about it being a template language, but that is exactly what you need!

[fivre]

JSON patches aren't the most intuitive and kustomize needs some helper tooling to generate them for you (given JSON objects A and B, generate a patch that transforms A into B), but overall the kustomize model makes more sense, and the team behind it seems to be more actively improving developer QoL stuff than Helm is

templates are only good if your templates can remain simple and do not need to expose most of the output fields. my experience developing a chart for wide distribution has been very much that your templates will not remain simple (and will turn into an incomprehensible mess, since you'll need them to handle tasks templates are fundamentally poorly suited for) and that there is always someone, somewhere, that needs some particular resource field exposed in values.yaml. the

> As a result, the number of possibilities for configuration is often unreasonably large and complicated, mimicking the actual resources they want to create, but without any schema validation!

bit from the op is incredibly true. values.yaml grows, over time, to have every field in the objects it generates, just organized differently, without validation, and with extra complicated relationships with other settings

kustomize allowing you to provide a base set of resources that users can apply their own patches to avoids that config surface bloat problem entirely

[Hamuko]

>Having the weird syntax for replacing things

Isn't the "weird syntax" just either Yaml files or just JSON Patches, which is a pretty easy standard?

>having to look at a bunch of different files to see what is going on

I consider that a feature, not a bug. prod/larger-memory-request.yaml makes it much easier for me to see what goes into deploying the prod environment instead of for example the test environment.

[cortesoft]

By "weird syntax" I mean stuff like "patchesJson6902" or "configMapGenerator" or "patchesStrategicMerge" where you have to know what each field means and how they work.

A template is much easier to read. I had zero experience with go templating, but was able to figure out what it all meant just by looking at the templates... they still looked like kubernetes resources

As for looking at a bunch of different files, if you like having a "larger-memory-request" file, you can still do that with helm... you can use as many values files as you want, just include them in precedence order. You can have your "larger-memory-request" values file.

[miiiiiike]

That just using Kustomize. There’s a difference between learning curve and frustration post learning curve. Kustomize isn’t that bad, Helm comes with far more headaches, especially if you need to do any kind of inheritance.

Keep your customizations flat and compile them to yaml+grep to find out what’s getting overridden and where.

[ahoka]

Do you have any really complex Helm template? It either becomes an unreadable mess or you end up with basically manifest snippets in your values file.

[ljm]

You shouldn't need to read the source to configure the values how you want.

Helm's problem is that `values.yml` is basically the API and every helm chart provides its own (often incomplete) interface with poorly documented defaults. Some of those can spread over 3k+ lines and it's utterly overwhelming to figure out what to do with that.

[candiddevmike]

There nothing better than a huge comment describing behaviors above a value: {} to ruin your day.

[cortesoft]

When I see something like that, I just search in the templates directory for where the value is used.

[morelisp]

> Probably even better is to ship a controller and a CRD for the config.

Maybe it's just us, but our operations team puts pretty hard restrictions on how we're allowed to talk to the K8s API directly. We can turn a regular Deployment around as fast as we can write it, but if we needed a controller and CRD update it'd take us like three days minimum. (Which, I even sort of understand because I see the absolute garbage code in some of the operators the other teams are asking them to deploy...)

[cassianoleal]

Generally speaking, operators and CRDs are more in the domain of your platform rather than your products. They should provide common interfaces to implement the business requirements around things like uptime, HA, healthchecking, observability, etc.

If a product team sees itself needing to deploy an operator, it's likely the platform is subpar and should be improved, or the product team is overengineering something and could do with rethinking their approach.

As in most cases, a conversation with your platform/ops/devops/sre/infra team should help clarify things.

[jen20]

If you run a multi-tenant Kubernetes cluster at scale, operators with poor discipline spamming the API servers and taking etcd down is a leading cause of sadness.

[morelisp]

This is the common view among our ops team, sure, but for a vocation so prima facie obsessed with postmortems/five-whys/root-causes/etc it's depressingly shallow.

[mountainriver]

Controller + CRD is the way to go and seems more in line with how k8s was intended to be used.

The challenge has historically been that controllers are a lot harder to write, but I think that story has improved over the years

[arccy]

operators are great when you control it. less so when it's some third party one that doesn't support that field you need on a resource it creates

and all the customizations just end up being yaml merges from a configmap string or CRD if you're lucky

[mountainriver]

Fair enough, the UX is just so much better that I'd gamble it in most use cases

[jpdb]

> Probably even better is to ship a controller and a CRD for the config.

But how do you package the controller + CRD? The two leading choices are `kubectl apply -f` on a url or Helm and as soon as you need any customization to the controller itself you end up needing a tool like helm.

[numbsafari]

Just use kustomize. It’s baked into kubectl. No need for a separate tool.

[cassianoleal]

Agree. I'd recommend to start with static YAML though. Use kustomize for the very few customisations required for, say, different environments. Keep them to a minimum - there's no reason for a controller's deployment to vary too much - they're usually deployed once per cluster.

[jen20]

The need to use something like Helm to distribute a complex application is a good indication you've built something which is a mess, and probably should be rethought from first principles.

Most of the problems associated with Helm go away if you stop using Kubernetes.

[kkapelon]

By that thinking "The need to use something like APT/YUM/DEB/RPM to distribute a complex application is a good indication you've built something which is a mess, and probably should be rethought from first principles."

So Linux is a mess? And we should rethink how rpm and deb work?

Or all Deb issues go away if you stop using Linux?

People forget that Helm is a package manager first and foremost (the only one for Kubernetes). It also happens to include a templating mechanism. The templating part has its issues, but until we find another package manager, I don't see Helm going anywhere.

[jen20]

> So Linux is a mess? And we should rethink how rpm and deb work?

Linux is indeed a mess, yes. RPM and Deb are both awful formats stuck in the 90s, with even worse package managers on the top. Even with the legacy of those, installing a package does not involve templating a whitespace-sensitive language with a mediocre template language.

> Or all Deb issues go away if you stop using Linux?

Never had any issues with debs on FreeBSD. Or NixOS if one likes the Linux kernel.

The idea that helm is even a package manager is fanciful at best, in any case.

[kkapelon]

> The idea that helm is even a package manager is fanciful at best, in any case.

The front page of helm.sh literally says "The package manager for Kubernetes". If it was advertised as "the best templating engine for K8s" or something similar I would agree with you.

People try to abuse Helm.sh as a fancy templating engine. And the testament to that is all the articles "Helm vs Kustomize vs JSonnet vs ..."

[jen20]

Lots of things describe themselves in fanciful terms. Helm has none of the trappings of a package manager, yet all of the trappings of a mediocre template renderer.

[imglorp]

Vendors shipping things for customers to run in their clouds and prems have a very limited set of common denominators. When you add in requirements like workload scaling, availability, and durability, that set is very small.

So yeah we do this. Our product runs in 3 public clouds (working on 5), single VM, etc. and our customers install it themselves. We're helm plus Replicated. AMA.

[Too]

Once you add in workload scaling, availability and durability, there is surely a dedicated ops team that want to control every aspect of how it’s deployed, including the security around it. They are not just going to blindly apply a chart without at least having reviewed it in great detail first.

What I found is that when doing such review, you realize 99 of the template variables are not relevant for you and the one place you need to template is missing a value. Just extracting the rendered manifests and modify them by hand from there becomes more maintainable. Like you say, there is a very limited set of common denominators.

For smaller orgs, just running a single container and increasing the Node size takes you a very long way. That doesn’t need helm.

[jen20]

This is absolutely the root of the problem. Most public Terraform modules suffer the same issues - configurable in so many ways it’s impossible to infer anything without a complete reading of the code.

[dog321]

When deploying into different clouds, do you require any cloud provider resources that require management with terraform etc. or is it relatively self contained?

Also curious what issues you've seen replicated prevent.

[imglorp]

For public cloud k8s, no we don't provision or TF anything, we just shove in a manifest and k8s creates the workloads and it provisions persistent volumes and load balancers on your behalf. That's either Helm or Replicated (Kots) on top of Helm. Yes, it's basically self-contained and manages to abstract most of the cloud differences. We do have a custom storage class for each cloud but probably don't need it. The network load balancers need a little cloud specific annotation.

Replicated saved work by handling a configuration gui for the end user, licensing/ entitlements, support bundle collection, private image proxy, things like that we didn't want to deal with.

[hinkley]

I barely have a horse in this race, but I think what I'd like to see is more apps that behave like 'npm config' or 'git config' where you can imperatively change one configuration value.

I take your image as the FROM for my own Dockerfile, tweak a few settings, maybe alter the CMD, and then run my image instead of trying to do some sort of ad absurdum variation on a Twelve Factor App.

[zzyzxd]

> helm is at it's best when you need to _publicly distribute_ a complex application

I would say, helm is at it's best when you need to _publicly distribute_ a complex application, AND the majority of the users on the receiving end don't care about the complexity.

You don't need Helm if your manifests are simple. But when your manifests become complex, helm will make it even more complex by turning each field in the plain k8s manifests into a toggle in a values.yaml file.

[pas]

but usually even the structure is wrong, and there's no option to disable a template in Helm, so I need to install and then manually edit/replace things, or just script Helm with bash or something, which is terrible .. so the whole things is just a big ball of WTF.

yes, it's nice to do env-var-substitution, and --set is not that dumb.