[renaudg]

Kapitan (https://kapitan.dev) is on my radar as a possible sweet spot between Kustomize and Helm.

Until now I've used Jinja2 templates for our Kubernetes definitions with a variables file for each environment, but this is awfully manual.

I'd love Kustomize to be sufficient for us as it's poised to become a standard thanks to now being part kubectl.

Unfortunately, in some ways its YAML patching philosophy is too limited, and coming from a templating system would be a step back even for relatively simple use cases : for example, you're very likely to need a few variables defined once and reused across k8s definitions (a host or domain name, project ID, etc). You can't really do that in a DRY way with Kustomize.

AFAIK, it also currently doesn't have a good story for managing special resources like encrypted secrets : it used to be able to run arbitrary helper tools for handling custom types (I use Sealed Secrets), but this has been removed recently for security reasons, prior to the Kubectl merge.

Kapitan seems to cover these grounds, and it doesn't carry the weight of those Helm features which are useless for releasing internal software, but I'm still a bit worried about the complexity and learning curve for dev teams.

Is there anything else out there that goes a little further than Kustomize, is simpler than Kapitan and Helm and fits well into a GitOps workflow ?

[markbnj]

> for example, you're very likely to need a few variables defined once and reused across k8s definitions (a host or domain name, project ID, etc). You can't really do that in a DRY way with Kustomize.

I agree this is one of the areas where you feel the pinch of kustomize's rather puritan design philosophy. We've been able to work around those things in ways that aren't exactly elegant, but don't cause physical discomfort. For shared variables we keep a patch on disk and generate specialized copies of it during deployment. It's a hack, but it retains some of the benefits of a declarative approach. We also still use substitution in a couple of places. It's hard to use kustomize to update an image tag that changes with each build for example.

[atombender]

I've only looked briefly at Kapita . It looks interesting, but I think what Helm gets right, and these other tools don't, is to have a real deployment story that developers can like. Helm doesn't excel here, but it's better than kubectl.

In short, I think the winning tool has to be as easy to use as Heroku. That means: The ability to deploy an app from Git with a single command.

It doesn't need to be by pushing to git. I built a small in-house tool that allows devs to deploy apps using a single command. Given some command line flags, it:

* Checks out a cached copy of the app from Git

* Finds the Git diff between what's deployed and current HEAD and pretty-prints it

* Checks the CI server for status

* Lints the Kubernetes config by building it with "helm template" plus a "kubectl apply --dry-run"

* Builds the Helm values from a set of YAML files (values.yml, values-production.yml etc.), some of which can be encrypted with GPG (secrets.yml.gpg) and which will be decrypted to build the final values.

* Calls "helm upgrade --install --chart <dir>" with the values to do the actual deploy.

The upshot is that a command such as "deploytool app deploy --red mybranch" does everything a developer would want in one go. That's what we need.

The tool also supports deploying from your own local tree, in which case it has to bypass the CI and build and push the app's Docker image itself.

Our tool also has useful things like status and diff commands. They all rely on Helm to find the resources belonging to an app, and we did this because Helm looked like a good solution back when we first started. But we now see that we could just rely on kubectl behind the scenes, because Helm's release system just makes things more complicated. We only need the YAML templating part.

I hate YAML templating, though, so I think something Kubecfg is the better choice there.

[markbnj]

> The upshot is that a command such as "deploytool app deploy --red mybranch" does everything a developer would want in one go. That's what we need.

That tool for us is a gitlab pipeline, and I guess the logic in your tool is in our case split between the pipeline and some scaffolding in a build repo. The pipelines run on commit, the image is built, tested, audited, then the yaml is patched and linted as you describe before being cached in a build artifact. The deploy step is manual and tags/pushes the image and kubectl applies the yaml resources in a single doc so we can make one call. We recently added a step to check for a minimal set of ready pods and fail the pipe after x secs if they don't come up, but haven't actually started using it yet.

[atombender]

That sounds similar, except you prepare some of the steps in the pipeline. Sounds like you still need some client-side tool to support the manual deploy, though. That's my point -- no matter what you do, it's not practical to reduce the client story to a single command without a wrapper around kubectl.

Interesting idea to pre-bake the YAML manifest. Our tool allows deploying directly from a local repo, which makes small incremental/experimental tweaks to the YAML very fast and easy. Moving that to the pipeline will make that harder.

Also, you still have to do the YAML magic in the CI. We have lots of small apps that follow exactly the same system in terms of deploying. That's why a single tool with all the scaffolding built in is nice. I don't know if Gitlab pipelines can be shared among many identical apps? If not, maybe you can "docker run" a shared tool inside the CI pipeline to do common things like linting?