[tn890]

I wonder what the need for tools such as this or other "Kubernetes-by-example" type pages tell us about the complexity of configuring Kubernetes resources.

Do we need a better layer of abstraction, i.e. better adoption and tighter integration for something like kustomize? Have we fucked up completely with Kubernetes due to it being outrageously complicated for simple tasks? How we redesign this to be simpler? Is the complexity even a problem for the target audience?

I've no idea. I just know I'm a kubernetes admin and I can't write a deployment yaml without googling or copy/pasting.

[kevinmgranger]

I think of Kubernetes as "the new Operating System", and these complex resources as fiddling with initscripts, fstab, /etc/interfaces, and so on. Writing an operator is like writing your own initscript.

I wouldn't be surprised if we eventually see new abstractions for "you just want a plain 'ol deployment with CI/CD du jour, a persistent volume claim, and a service ingress, just like 90% of all other CRUD webapps? Sure, here's a simple resource for that."

I think we'll start seeing a move towards more "opinionated" tools, just to outsource some of the decision making. No sense learning how to write your own pipelines if you can find a tool that says "we're gonna deploy every time you make a git tag and run `mvn package`, you figure the rest out".

[oauea]

Helm has been doing this since the early days. 99% of our charts are created using:

* `helm create` to get the default scaffold

* modify a handful of entries in the generated values file

* done!

Only thing is the default helm chart starter does not allow for autoconfiguring of volumes, and since we're porting a lot of stateful apps to kubernetes we just modified the default starter to include that capability.

Of course it would be nice to not have to maintain a bunch of different virtually identical templates.

Pulumi looks interesting but unfortunately seems to insist on vendor lock-in (see the jerk-around on https://github.com/pulumi/pulumi/pull/2697). So I'm looking forward to the AWS CDK (https://aws.amazon.com/cdk/) maturing a bit.

[jwatte]

Helm is an example of what I mean by "pusing against the boundaries." We tried using helm, but we kept running into limitations, and the complexity of trying to customize charts that didn't work right ended up being worse than the original problem. Currently, we're using kustomize, plus a thought out manifest hierarchy, which is working out better.

[wronglebowski]

As someone attempting to port a docker-compose application to a Helm chart I'd love to hear more about the resources you used. As someone working with a very simple application I've found the processes to be more difficult than I anticipated.

[oauea]

Kubernetes will be far more complex than docker compose. It'll let you do much, much more also.

Some potentially useful tips:

* One docker container usually maps to one pod, which is created by a deployment. You can put multiple containers in a single pod, but this couples them together tightly.

* Use services to assign hostnames to pods

* Have pods communicate to each other using the service names. This works the same as putting them in the same docker network.

* Volumes depend on your cloud provider or if you're running on bare metal. In the cloud it's easy, you just request one and it gets created on demand and backed by a cloud disk.

* If you're using volumes, you probably want to use the Recreate updatePolicy for your deployment. This will ensure the old pod is shutdown before creating a new one. Which is necessary to work with block volumes.

When using helm start with a `helm create CHARTNAME` and take a look at what it generated for you. You'll get some heavily templated yaml that if you're lucky you will barely have to touch.

But it's best to go through these tutorials and learn how to use the basic building blocks of kubernetes directly: https://kubernetes.io/docs/tutorials/kubernetes-basics/

Once you're familiar with what a Deployment, Service, Ingress and PersistentVolumeClaim are you can use helm to template this for you where necessary.

[skzo]

You can give a shot at kompose (https://github.com/kubernetes/kompose) It will convert your docker-compose.yml to kubernetes automatically

[abraxas]

And the tower of ever more obscure abstractions just keeps on growing. Until one of these fragile layers starts to malfunction and the whole thing comes tumbling down and nobody has a clue how to fix it all up.

Collectively, as an industry we have gone insane.

[harpratap]

People have attempted simpler abstractions and it has always failed. CloudFoundry, Mesos+Marathon, Heroku were all simple at first and none of them revolutionized the industry.

[oso2k]

I think of writing Operators more as writing your own Device Driver but I can see the correlation you're trying to make.

[taywrobel]

Well prepare to be unsurprised! The CI/CD abstraction already exists in Tekton - https://cloud.google.com/tekton

The rest are soon to follow I’m sure.

[jml78]

If people think deployment yamls are complicated, what til they see tekton.

Don’t get me wrong, I actually love tekton because I hate everything that is Jenkins and most other CICD tool integration with k8s. So tekton is just amazing in that regard.

But there is a huge learning curve for people who are used to old school tools like Jenkins and its pipelines .

[person3]

Agreed. Where I work we have a core team that created a framework that manages the Kubernetes configuration for us. As a backend developer I basically just have to throw a config file into my service with some basic options like which port to listen on and where the docker container is, and then it gets deployed with everything setup for me (using helm I believe).

I really hope we move more towards these opinionated tools that can handle 90% of the use cases. Most people just want to host an app on a port, and it's a pain to have to develop that pipeline at every company that wants to adopt Kubernetes.

[sushshshsh]

Agreed. Just like how I don't want to roll my own desktop environment on Linux, I will happily accept a good config rollef for me, even if opinionated.

[throwaway894345]

I've always felt that Kubernetes isn't appropriate for most businesses to use directly, but rather it's a platform for simpler platforms--someone like Heroku would build on top of Kubernetes and expose a much simpler interface for their users so they don't have to think about SSL, DNS, logging, load balancing, auto-scaling, etc. Alternatively, maybe the problem is solved with "distributions" of Kubernetes analogous to Linux distros--preconfigured Kubernetes installations so you don't have to figure out how to configure your own Kubernetes service (arguably this is also what cloud providers give us with their managed k8s offerings). I'm curious to hear what others think.

[linkdd]

100% agree. Kubernetes alone is just a """framework""" to describe your infrastructure, definitely not an "end-user product".

That is why in many businesses there is an OPS team managing the Kubernetes and providing tools like Cert-Manager, Istio, ... and the rest of the company who just use what the OPS team made.

Right now, everyone is building its own distro, proving IMHO the need for it.

[throwaway894345]

Well said!

[theptip]

Lots of PaaS offerings have been doing this for a long time, RedHat OpenShift being the original one; they contributed a lot of 1.0 k8s code and provided a lot of key early architectural feedback by implementing a PaaS using k8s. Another early one was Deis, who were at the first KubeCon.

All this is to say that you're at very least certainly half-correct, in that k8s is a very flexible tool that can be used to build a very simple, elegant, and ergonomic PaaS.

I'm not sure I agree that it's inappropriate for most businesses though, unless you think that only a PaaS like Heroku is appropriate for most businesses; the analogy I'd suggest is "Heroku vs. running your own VMs" circa 2010. Heroku is great for getting started, and lets you move fast by abstracting away a bunch of infra. But it's also restrictive; you can't pick and choose your components freely. As you grow past a certain point you'll almost certainly need the flexibility (or just cost-effectiveness) that you get from running your own infrastructure.

K8s is an improvement here because you can run a managed cluster on something like GKE, which takes away most of the operational toil, while still giving you a lot of flexibility on what components / pieces to include. The k8s domain API does a great job of abstracting away true infra concerns like volumes, compute, scheduling, load-balancing, etc, while making it really easy to package, use, and iterate on the stuff that sits on top of that infrastructure.

I'd probably not encourage a seed-stage startup to use k8s unless you're very familiar with the tool; a PaaS like Heroku would likely be more appropriate. However at the point that you'd usually graduate to running your own VMs (wherever that is on your own company's developmental path), I'd say that using k8s is now a better choice.

[throwaway894345]

I think I generally agree. Maybe saying "most businesses" is a poor word choice; I don't really have the breadth of experience to speak for the majority of the industry; however, I do strongly suspect that a lot of companies are employing an ops team to wrestle with Kubernetes when their needs could probably be met by a PaaS, and I think this will only get more true as the first generation of PaaS-on-k8s arrive on the market in mature form. I generally agree with this:

> However at the point that you'd usually graduate to running your own VMs (wherever that is on your own company's developmental path), I'd say that using k8s is now a better choice.

However, even then there are intermediate options between VMs and full-on k8s, such as AWS ECS/Fargate, a Kubernetes distro, or managed Kubernetes offering (e.g., GKE) which give me the flexibility to interact with k8s, but they come with sane, pre-configured (or easily configured) solutions for logging, monitoring, ingress, load balancing, upgrades, etc.

[joshmanders]

That's exactly what I am doing with https://primcloud.com :)

[throwaway894345]

This seems really cool; I was poking around, but half of the links in your footer are broken (presumably placeholders for future pages?).

[joshmanders]

Yes sorry, I had a simple splash landing page up but people were asking for more info, so I put this up while I continue building the infra/platform.

I will over time expand on the content, but I'm focusing on the platform itself over marketing stuff.

[andendau]

Do you support on-premises kubernetes?

[joshmanders]

I'm still building so I currently don't. Kubernetes is just a tool I use to deploy my customers sites on, it's not the selling point.

I do however have plans of supporting enterprise by packaging a version of the app into an on-prem deployment system so you can just deploy your own kubernetes cluster and install the app and have your own enterprise version of Primcloud.

[erohead]

https://kubesail.com offers the exact service you describe!

[erulabs]

Thanks for the shout-out! We’re trying to be like an “open-Heroku” - back when I worked on OpenStack it was called “the open cloud” and no one knew what we were on about - open stack was enterprise as enterprise gets!

Our goal is to make a Heroku-ish platform for getting an app online - but one that doesn’t hold you over a barrel later on. You can even host from a spare home server :)

(Disclosure: I’m the CTO)

[OJFord]

Do you still expose the k8s bits?

I'm interested in/have previously given up on a product where the control plane is managed for me, I can join the cluster with bare metal nodes, and then it's just Kubernetes.

[erulabs]

Yep - we expose the Kubernetes API (and add mutating controllers to help automate some of its actions).

Right now we don't do "hosted control plane", rather - you can either use our hosted clusters where you have namespace-level access (so you still get to use Kubernetes, just not all of the resources, for example, no `Nodes`), or you can attach your own cluster to our UI (with the advantage that we can setup ingress and forward traffic to you, which is perfect for a home-hosting setup).

Hosted control plane is something im watching closely though - I'd really love to offer that as a service, but since we're small, we're trying to focus hard on a core offering. Will be considered in the future though!

[kutorio]

kubesail has really nice UI for an hosted option. Another alternative I found is https://kalm.dev/, which is open sourced and geared towards configuring a heroku on top of a k8s cluster.

[manigandham]

Complexity always exists somewhere. You can't remove it, only try it make it easier to deal with.

What Kubernetes allows you to do is very complex so you need a capable system to express it all. YAML isn't always the best but it works fine for most and tools like these are very helpful.

Do you use an IDE? Does that mean the language and framework is too complex? No, it's just a tool to help you get things done. More tools aren't a bad thing.

[gambler]

>Complexity always exists somewhere. You can't remove it, only try it make it easier to deal with.

This is simply not true. Most complexity in today's systems is completely avoidable. Most developers are just mentally stuck and don't even try. "Managing" complexity is a great way to achieve job security without becoming good at anything specific. Instead of learning how to design system people are learning how to write config files.

[manigandham]

That's a different argument. Not needing it doesn't mean it doesn’t exist.

If you don't need Kubernetes then don't use it. But if you do then you can't make it any simpler than the complexity needed to deliver the functionality.

Built in deployments, process health, logging, load balancing, security, high availability, config and secret management, volumes and persistence, and much more in exchange for some YAML files is a pretty good encapsulation of complexity though.

[closeparen]

Needing an IDE to have a reasonable time is a very common criticism against the expressive power of certain languages.

[enriched]

Agree with this, and would say that often a good way to deal with complexity is encapsulation. I imagine there are some developers that say "why wouldn't you just write it in assembly" Because if you are used to that and write small low level binaries it might work well. But trying to write a larger app it becomes more difficult to reason about.

[haolez]

Not all complexity is intrinsic, but all complexity incurs a cost.

[xyzzy_plugh]

Most of my colleagues find the yaml interface to be a lost cause. I tend to prefer to use the gRPC API as it's effectively the same declarative operations but you get to control it in software, which just feels right. YAML is just a crutch for specifying the API calls the client needs to make.

[EdSchouten]

You mean the Kubernetes HTTP API? That one isn’t gRPC based, right?

[xyzzy_plugh]

Ah right, guess I was thinking of the protobuf over HTTP API. In any case, I find using the protobuf types in your language of choice pretty ergonomic compared to YAML.

[gen220]

This is one thing that mesos/aurora got right (the "turing-complete-by-design" config files, written in Python). Templating yaml is like templating html. It works but absolutely sucks at scale if your job is to maintain it.

It's good to know that such an interface could conceivably be built for Kubernetes, using the HTTP API. Even better that it supports any programming language (since it's a network interface); I'd see that as a step-up from aurora configs, in fact.

At some point, I imagine this will become the default way that Smart Organizations build around k8s, once k8s's core API is super-stable and people develop sound frameworks in $POPULAR_LANGUAGE for declaring your configuration.

Then, we can then banish yaml to the same fate as json and xml. One can dream, at least. :)

[xyzzy_plugh]

> the "turing-complete-by-design" config files, written in Python

Funny you mention this, check out what is effectively this but for Kubernetes:

https://github.com/stripe/skycfg

[gen220]

Wow, this is great! Thank you for sharing.

SMART_ORGANIZATIONS=$SMART_ORGANIZATIONS:stripe

I wonder if they were inspired by aurora, it's very similar.

[teqnologiq]

Would you happen to have any links to Github repos that would be an example of this?

From light googling I only see examples of building gRPC APIs and microservices.

[xyzzy_plugh]

I don't have an example offhand, but mentioned skycfg in the neighbor comment, which may be of some interest to you. I've used https://github.com/ericchiang/k8s in the past with great success.

[wutsyrpt1]

Is it anymore complex than all the old ways?

Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier?

I seem to remember a lot of wrangling custom kernels to get Asterisk sounding just right, bizarre Apache, & network configs.

It’s just text? It’s always going to turn into a nebulous mess without literal edges and boundaries.

That’s Google’s play with it, IMO. Train tracks. Which is what I hate about it.

Google hasn’t built a less Byzantine text mess. It’s built hype though, with a boring tool

[ex_amazon_sde]

> Is it anymore complex than all the old ways?

> Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier?

Yes, and by far. Adding a layer on top of all the traditional Linux daemons, tools and libraries does not decrease the total complexity - quite the contrary.

When you have a bug in an application that is related to something in on another layer you have to walk through the whole stack.

Examples: A bug in a network card impacting only large UDP packets. A race condition of file access triggered by NFS or a storage device driver. A vulnerability based on a timing attack due to CPU caches.

The deeper the stack, the worse.

[doteka]

But wouldn’t the point be that you don’t care about hardware level problems anymore? When I find a node with issues, I can just delete it from the pool and get a fresh one back. The bad network card? That’s for Google/Amazon/DigitalOcean to deal with.

I find the bog-standard Prometheus chart provides me a pretty incredible level of monitoring out of the box, usually it’s pretty easy to pick the bad one out of a graph.

Running your own VMs without something like k8s? Yeah this setup I can deploy and have working in an hour is gonna take you a week to set up properly. Standardization is valuable. Abstraction is valuable.

[ex_amazon_sde]

> But wouldn’t the point be that you don’t care about hardware level problems anymore?

No. Read my post again: I did not wrote about hardware issues.

Most work around optimization, reliability or security require digging through the whole stack sometimes down to the kernel.

> When I find a node with issues, I can just delete it from the pool and get a fresh one back.

However, a lot of k8s deployments are on-premise, where you have to debug your own hardware.

> The bad network card? That’s for Google/Amazon/DigitalOcean to deal with.

First you have to pinpoint the root cause of that glitch affecting all the containers running on VMs using the same bad drivers. Often it could be the same in 50% of your fleet.

> is gonna take you a week to set up properly.

Most certainly not. I've been deploying large production fleets in minutes since 2005.

[adwww]

There are plenty of online tools to generate Nginx or Vagrant config files.

I don't really get the love for hating on K8 complexity on here - nobody says it's the perfect tool for every use case... but when you do need it, it's amazing, and has many advantages over the current trend for serverless IMO.

[echelon]

> I can't write a deployment yaml without googling or copy/pasting

What about Nginx or Apache configuration files? Could you write them without googling?

[goatinaboat]

What about Nginx or Apache configuration files? Could you write them without googling?

Easily, because noone writes them from scratch, they just modify the default one.

For most of my Kubernetes work I "kubectl describe" something existing into YAML, modify it, then "kubectl apply" it back again.

[marcc]

You can do this without a cluster also:

  kubectl create deployment my-deployment --image nginx --dry-run -oyaml

[holografix]

I find K8s to be the perfect tool for enterprises.

It provides the promise of portability, so the business feels less locked in to a vendor, while still being dense and impenetrable enough to them that IT won’t be done out of a job.

Simpler, more efficient alternatives are seen as more expensive, because IT hides the true cost of just how much of their salaries is spent of non-productive pottering around with yaml.

These systems and products also threaten to automate IT out of a job and don’t necessarily count much for their resumes. If you’re not a programmer what interest do you have in dumping all of your intricate, fragile environment for a PaaS? Not much

[jayd16]

I like reading declarative docs because everything is spelled out but I hate writing them for the same reasons.

More tooling to write good k8s yaml sounds good to me.

[renewiltord]

I always commit an example k8s YAML to any repo with k8s in it that describes how to get 2048 running.

Honestly, I get the pieces and I know where to look to get what so I'm not too bothered. The problem with wrapper tools is that sometimes I can't get at the insides and then I have to learn the wrapper tool. So I'm going to just stick to raw Kube until one of the wrappers wins out.

[snom380]

Isn't that the case with any tool or service of some complexity? I'd probably have to look at docs or examples for most config files i write.

There's trade-off between a tool being too specific to a usecase vs being too generic with the associated "boilerplate". But I don't know how much simpler Kubernetes could be made while still covering the intended scope?

[p_l]

I would say kubernetes is very simple, in the same way lambda calculus or composing programs out of pipes is simple.

It has very simple basic model, which allows you to recursively built more and more complex abstractions on top of it.

[secondcoming]

IMO it needs two things:

- a dedicated editor with intelligent autocomplete

- stop using YAML, it soon becomes unreadable. JSON is easier to grok.

[goatinaboat]

JSON is easier to grok.

For anything non-trivial you will want inline comments.

Also while any JSON can be expressed in YAML, the reverse is not true.

[reificator]

> For anything non-trivial you will want inline comments.

Yes, JSON needs comment support back.

> Also while any JSON can be expressed in YAML, the reverse is not true.

This is a feature not a bug.

[p_l]

Unfortunately, JSON is much more annoying to edit.

Best way is to stop using both, and generate the objects from higher level language, at least something like Jsonnet (which is really just a step up, so better not stop there but I will take what I can)

[FridgeSeal]

> generate the objects from higher level language

I'm on the fence about this, something like Dahl seems good, but I am not a fan of using fully-fledged higher level languages (the Pulumi approach) as I've been burnt before with devs writing bad code and generating configs in the most confusing, convoluted way possible.

[reificator]

> a dedicated editor with intelligent autocomplete

A.K.A. a schema file that a general purpose editor can consume. Please don't make me use some single-purpose editor just for autocomplete.

[marcc]

Kubernetes schema files and a way to run them can be found at https://www.kubeval.com/

[crazy_hombre]

Isn't YAML supposed to be a superset of JSON? So technically, you should be able to use JSON files right now (barring a few exceptions).

[gk1]

It's a never-ending cycle. See: Front-end development. Until yet another abstraction emerges, anyone using Kubernetes for mission-critical applications should be using automated testing to prevent awful outcomes from a single botched line of YAML. There are a bunch of solutions emerging in this space, like Datree[0], made for the sole purpose of preventing Kubernetes misconfigurations.

[0] https://datree.io (Disclaimer: I work with them.)

[jwatte]

"Simple" solutions to "complex" problems, by necessity, only work for certain subsets of the problems. Modern distributed systems have tons of largely essential complexity. The only way to reduce it, is to reduce the expressive power of the solution, which might be great for the cases where you fit into the mold, but then fails spectacularly (either with excess resource consumption, or by not working at all) when you push against the boundaries.

[auspex]

I think abstracting YAML generation to a tool is a good idea. Let the tool handle the underlying abstraction of converting the parameters to YAML.

If k8s changes notation having a layer of abstraction (hopefully) allows me to run the same command to generate new YAML (very helpful in automation)

[runbsd]

A lot of other "bleeding edge" technologies that blow up and become difficult to use end up with new abstractions, and then abstractions for the abstractions, and the original problem the tool was meant to solve is now not the focus any longer.

[drybjed]

https://xkcd.com/1319/

[runbsd]

there is always a relevant xkcd!

[jrockway]

I don't think better abstraction is going to help the underlying problem: there are a lot of questions to answer, and you don't know the answer.

Without going into the complexity of Deployments, consider the lowly Pod. What configuration does your app need? What is the name of the container that contains it? How much memory does it use? How much CPU does it need? What ports does it listen on? What HTTP endpoint handles the health check? Does that endpoint test liveness or readiness? What filesystems does it need? What setup needs to be done before the main container runs? Does it need any special resources like GPUs? The list goes on.

The problem here is that when you're writing a Pod spec, you're building a single-purpose computer from scratch. In the traditional UNIX world, people answered most of these questions for you. How much RAM can my app use? However much I plugged in. How much CPU can my app use? All of them. What ports does it listen on? Any of them from 1024-65535. What filesystems does it need? Whichever ones I setup in /etc/fstab.

I don't think it's a stretch to call UNIX's "yolo" approach problematic. It is great when you have one server running one app, but servers have gotten gigantic (with pricing to match) while applications have largely stayed the same size. This means you have to pack multiple apps onto one physical server, and to do that, there have to be rules. When you write a Kubernetes manifest, you are just answering every possible question upfront so that the entire system runs smoothly even if your individual component doesn't. It's the cost of having small apps on big computers.

The problem comes from applications that you didn't write, or don't fully understand. Before you can understand how the application behaves, you have to write a manifest. But you don't know the answers to the questions like how much CPU you're going to use, or what the worst case memory usage is, etc. This causes a lot of cognitive dissonance, because the entire file is you admitting to the computer that you have no idea how to configure it. No abstraction layer is going to fix that problem, except by hiding those uncomfortable details from you. (And you will always regret using the "yolo" defaults -- who hasn't tried to SSH into a broken server only to have Linux helpfully OOMKill sshd or your bash instance when you're just trying to kill your malfunctioning app.)

This is largely the fault of application developers. They aren't willing to commit to reasonable resource limits because they don't want to handle support requests that are related to underprovisioning. My experience is that applications that set limits pick them wrong. For example, GCP and DigitalOcean's managed Kubernetes offerings both install monitoring agents to support their dashboards; these apps ship with limits that are too low and any reasonable Prometheus installation will notice that they are being CPU throttled and warn you about it. Now you have to waste your day asking "is this a real problem?"

Many open-source apps go the other way and pick resource limits that truly encapsulate the worst case and require individual nodes that are many times larger than the entire cluster. Yes, it would be nice if I gave each pod 32 CPUs and 128GiB of RAM... but I don't want to pay $2000/month/replica thankyouverymuch. (I've been on the other side of that where resources didn't cost me real money and happily used terabytes of RAM as cache.)

Application-level configuration is also not in a great state. Everyone tries to sell you their curated defaults so they don't have to write any documentation beyond a "quick start". (I'm as guilty of that as anyone in fact!) The application will have some built-in defaults (so the developers writing the app can just "go run main.go" and get the config they need). Then someone comes along to make a Helm chart for you, and they change the defaults so that their local installation doesn't need any customization. This only causes problems because instead of an undocumented underlying application, now you have that AND an undocumented abstraction layer. You may find the answer to your question "how do I configure FooApp to bar?" but have no way of communicating that config through the Helm abstraction layer because the author of the Helm chart never thought anyone would do that.

This rant has gotten quite long so I'll wrap it up. No abstraction layer is ever going to make it so you don't need to answer difficult questions. The actual list of questions to answer is available through "kubectl explain pod.spec" and friends, however.

[dpedu]

In my opinion, Kube's yaml blobs are comparable to binary programs. Complaining that an executable has its settings, paths, ports, memory limits, etc, hard coded inside of it is an obvious code smell.

Tools like Helm solve this problem.

[jrockway]

Abstractions have a cost and a value. Programming languages are more expressive than writing out binary opcodes by hand, but don't really remove any features from the underlying machine. If your machine can do something, your C program can probably do something. So we use C. (It even ADDS the value that you can describe a procedure without knowledge of the underlying hardware. It may not be optimal on every machine, but at least you have a starting point for optimization!)

Helm, like any abstraction, has costs and values. For example, it's very valuable to be able to encode something like "my app consists of a frontend and a backend that run in the same Pod and must each use the same version of the code":

    apiVersion: v1
    type: Pod
    metadata:
      name: foo
    spec:
      containers:
        - name: frontend
          image: my.registry.io/foo-frontend:{{ .tag }}
        - name: backend
          image: my.registry.io/foo-backend:{{ .tag }}

Now your frontend and backend containers can't get out of sync, and it saves someone from having to manually ensure that they are sync'd. There's no way to do it wrong! You supply {{ tag }} and it remembers the constraint that you required.

The problem with this abstraction is that there's no escape hatch. If you wanted to run different versions of foo-frontend and foo-backend, there is no way to say "but no, really, this time I'm violating the rules for a good reason". You've reduced the features available... the only way forward is to start over with nothing.

The result of this is that every individual Helm chart has to account for every possibility that the manifests could possibly encode, and invent their own programming language that is identical in expressiveness to the underlying manifests. And they do! Differently every time! For example, if I wanted to allow people to override the container images, I'd have to make my template look like:

    containers:
      - name: frontend
        {{ if .tag }}
        image: my.registry.io/foo-frontned:{{ .tag }}
        {{ else }}
        image: {{ .frontend_image }}
        {{ end }}
      - ...

Now it's possible, but not in a way that anyone could search for on the Internet. You will have to read the code or hope I wrote documentation for my ad-hoc Kubernetes extension.

I think we can all agree that didn't save anyone any time or effort.

This example conveniently flows into my other complaint with Helm. The "yaml files" that declare templates aren't actually valid YAML. You can't use something like `prettier` to autoformat them. You can't use the YAML language server to provide code advice as you type. You can't use `kubeval` to validate them. You are throwing all of that away to Build Your Own Thing. It is actually very insidious and for that reason I consider Helm to be more harmful than helpful. It isn't an abstraction, it's just a macro that might be good for one person the instant they happened to type something in.

The other problem is that Helm charts have no upgrade path. They are only designed for "please explode this project into my cluster, I promise to clean it up In The Future". It never gets cleaned up and brings a little piece of un-updated Windows 95 nostalgia right into your cluster.

Helm is actively harmful. And people love it, because it saves them a tiny bit of time one day at the cost of a lot of time in the future.

[dpedu]

Templating within yaml is an absolute mess, I harped on ansible in the past for the same reason. However, dealing with that once is far, far better than dealing with thousands of hand-maintained kube yamls multiplied by your number of environments. Helm of course has its own pain points, but that doesn't subtract from the fact that it solves problems.

I see putting your example of two, unrelated, containers in the same pod as the same binary problem I mentioned earlier. I get your point, but it's a scenario one must wedge themselves into by making other poor choices. Why must the frontend and backend be the exact same version? The most obvious possible reason could be that the API used between them isn't versioned. Or maybe there's not even an API!

> You will have to read the code or hope I wrote documentation for my ad-hoc Kubernetes extension.

No, helm's approach to inserting variables into templates means this isn't the case. Every option and default appears in values.yaml and it's a one stop shop to see everything you can customize. The code example you wrote could be better written as:

      containers:
        - name: frontend
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

With values.yaml having

    image:
      repository: my_default_image
      tag: my_default_tag

Note that values.yaml is part of the template and values passed to helm's cli can override individual values in it.

I'm not sure what you mean about helm charts having no upgrade path. IME you can un-deploy, upgrade, and rollback helm deployments and it takes care of adding/removing kube resources that where also added or removed in the yaml for you. [1]

[1] https://helm.sh/docs/helm/helm_rollback/

[YarickR2]

what's wrong with googling or copypasting ? toolset (helm template set) with basic blocks is added rather quickly, after that it could be as simple as running helm oneliner with needed variable values in the command line

[spullara]

Open Stack by Google.