[guslees]

The fork is my fault. I agree it's confusing.

I wrote (and still maintain) kubecfg. Heptio joined the project and started adding lots of great stuff. Eventually it was clear they wanted to take it in a direction that was different to the original borgcfg-like vision. I suggested we split that new functionality into a different tool so we could keep exploring both directions without trying to mash both into the same cli flags. Hence ksonnet/kubecfg and ksonnet/ksonnet.

They both use jsonnet internally, generate the same k8s resources in the end, and have that common code heritage, so have many similarities.

ksonnet/ksonnet has a bunch of extra ("rails-like") tooling to hold your hand while you generate jsonnet, and assumes it uses the k.libsonnet library (https://github.com/ksonnet/ksonnet-lib).

ksonnet/kubecfg is much dumber and really just conceptually `jsonnet | kubectl apply`. In particular kubecfg avoids having any opinion about which jsonnet libraries you use. We use it extensively with https://github.com/bitnami-labs/kube-libsonnet, but you can also use it with https://github.com/ksonnet/ksonnet-lib or anything else that is valid jsonnet.

[atombender]

Thanks for the explanation. Is Kubecfg like Helm in that it will apply a "destructive" diff (i.e. delete resources belonging to a deploy that are not part of the new deploy)?

I took a look at Ksonnet a while back, and it seemed to have way too many bells and whistles. In particular, I did not particularly like the sheer amount of files, in something like three separate folders, that you have to maintain for each project. I love the idea of declaring a schema that you're generating data for, but Ksonnet seems to have adopted a fairly complex structure to accomplish this.

We use Helm right now, though purely as a templating engine and destructive deployer of charts that we store in the same repo as the app itself. The package management isn't useful for our own apps, and really only gets in the way. We're looking at alternatives that don't use text-based templating and come with slightly higher-level concepts of releases.

Right now, we wrap Helm with our own little CLI so that we can, for example, automatically build all the template inputs from a set of (defaults, environment-specific stuff, local overrides). Our tool also presents a diff if you want, and records things like git revision info and deployer name in the annotations. Thus, when you ask our tool to deploy something, it can look at what the currently deployed revision is, then do a "git log" to find what's changed, display that history with nice colours on the terminal, etc. before deploying. All things that, in my opinion, a Kubernetes deploy tool should have.

[ithkuil]

(EDIT: the nick rang a bell: thanks for ktail!)

kubecfg has the --gc-tag flag, which you explicitly pass so that it can know which of the existing resources in the k8s API server used to be created by this set of kubecfg maintained config (this "system", this "application", it's up to you to decide the grain of your modelling) and thus be able to delete the resources are are no longer output by the evaluation of the current configs.

This catches the cases where you delete resources but also when you rename resources, or when you "move" them between namespaces.

It's implemented without any in-cluster component (no "tiller") by simply setting the GC tag as a label on the resource.

Kubecfg also implements diff between local config a d deployed state.

As for the amount of files: it's up to you. Kubecfg is not opinionated on how you lay out your config.

Unlike helm, it doesn't require you to know in advance which values you want to parameterize (and thus out in a values.yaml) since it's trivial to override any value with jsonnet.

The k8s API and its data model is the only thing you have to learn.

There are some helpers available to help you structure larger configs, e.g. https://github.com/bitnami-labs/kube-libsonnet/blob/master/k... . While this lib also provides some 'macros' to help you build common entities like services and deployments, IMHO the main benefit is the mapping of "foo_: {key: val, ...}" to "foo: [val, ...]". The former is much more friendlier when you have to override values with jsonnet rather than having to depend on array ordering.

[atombender]

Awesome explanation, thanks! (And I'm really glad you like ktail!)

This sounds like a much better foundation to build on than Helm. To me, Kubecfg sounds a lot more palatable than both Helm and Ksonnet.

As an aside, Kubecfg still isn't something you (or at least I) would expect developers to work directly with. We have devs who currently just do "tool app deploy foo" (using our homegrown CLI) to deploy an app; they don't need to understand much about Kubernetes, though they understand the basics about pods, kubectl and so on. With Helm, they'd need to know to run "helm install --upgrade --values-from k8s/production.yaml ./chart" or some other extremely long command line. In short, none of the existing tools are high-level enough.

There are Heroku-like PaaS abstractions on top of Kubernetes that give you a simplified entrypoint into deployment, but I feel like what's needed isn't a whole platform, just an opinionated top layer. Kubernetes deals with discrete objects, what you want is a higher-level tool that deals with atomic groups of objects, i.e. apps.

Long story short, are there any rumblings in the community about going in this direction? The lack of such a tool, at least as far as I've found, has lead me to consider maybe creating one, based on the experience we've had with our in-house CLI tooling, and perhaps using Kubecfg as a foundation. Thoughts?

[guslees]

Making an "easier" app-level experience very quickly becomes (necessarily) opinionated, because you need to anticipate which k8s parameters need to be exposed and which can be derived/assumed. Narrowing the configuration space in this way is entirely the point of "easier".

The way I've been approaching this is that you need a local "power user" who produces a simple abstraction that captures local patterns and policies, and the rest of the company then reuses that abstraction (or abstractions). Helm sort of lets you build this, but in practice it requires re-packaging helm charts with local customisations - which rapidly becomes a lot of overhead. The alternative is to expose every possible k8s option through the original helm parameters, which in turn means the helm chart becomes bewilderingly complex, and we're back to our original problem statement.

Instead, I've been advocating an "overlay" approach with jsonnet and the design of kube.libsonnet. The idea is that each consumer can import some upstream k8s manifests (described in jsonnet), apply some further jsonnet-based translations, and then publish the result as newer/simpler "templates". Someone else can then consume that, add another layer, republish, rinse, repeat. Importantly, each "republished" layer is still as easy to consume as the original. Eventually you end up with a jsonnet-based template that becomes highly opinionated and specialised to your actual problem domain, and hopefully is terse enough for local devs to use without having to learn all about k8s.

Example strawman:

  local mycompany = import "mycompany.libsonnet";
  mycompany.PhpApp {
    repo: "webgroup/guestbook",
    url: "https://mycompany.com/guestbook",
    requires_mysql: true,
  }

This might (hypothetically) turn into a:

- k8s Deployment that derived the docker image from the repo name (using knowledge of local build/publish conventions) and the command from the fact it was a php app

- k8s Service to point to the Deployment

- k8s Ingress from the provided URL (and local policy), pointing to the Service

- Bring in a mysql instance via any one of several approaches (eg: new standalone instance, or configure a new user/table on a centrally-managed DB server)

None of the above would be hard to do right now using kubecfg (or other approaches), but requires at least one person who understands both local policies and kubernetes - and for them to express that knowledge in "mycompany.libsonnet".

Importantly, whatever "mycompany.PhpApp" did would be quite different to "mycompany.PeriodicSparkJob" or "someothercompany.PhpApp" - so this isn't really something the _community_ can provide, without it rapidly becoming generic again and missing the whole point of the exercise. Coming back to your question, I think this is why you won't (and will never in the general case) find already-made tools that just happen to match your particular local needs.

[atombender]

Those are some great points. I agree with the premise that Jsonnet and schema-based config generation opens up the possibility of actually composable, "layerable" building blocks, something Helm doesn't do at all. I also see your point about the top layer being org-specific.

That said, I was actually thinking more about the CLI itself, and wrapping the underlying config generation in something that, for example, knows how to tag the config (so that, if it uses Kubecfg internally, --gc-tag is automatically provided, for example. And using git as a base for release versioning. As I mentioned earlier, one thing our internal tool does on deploy is to present you with what commits will be deployed, which is derived from running "git log HEAD..<currently deployed commit>". It's a nice UX for the person doing the deploy. It just uses Kubernetes annotations for that, but it ends up being pretty powerful. Something we were also thinking about was using a CRD to record each deploy, so that you can get a history, with what commit, who deployed, and so on.

Another thing we do is provide a real-time progress view of the Kubernetes resources that your deploy creates/updates/deletes. This lets the operator know when the new version is live, and also alerts them if the deploy failed. Again, it's about UX.

I think I'd want to extract what we have into a general-purpose tool, and use something like Kubecfg or Ksonnet to do the actual applying of configs. But I don't hear a lot about what people are using, and looking for, in terms of deployment tools. For me, creating an in-house tool like this was an obvious thing because we just can't run kubectl or Helm from the shell to do things, it would be way too many steps even for simple apps. Is everyone writing tools like this? Or are they actually writing out full "helm install" commands?

[ithkuil]

A point on flags: yes it would be great if the user wouldn't have to remember to provide --gc-tag explicitly. Bringing that even further, I'd like to be able to specify the cluster in the config (likely in the last "actualization" layer). Conceptually it's like the namespace: you can currently craft configs that are parametric on a given namespace and then fix that value to a given deployment specific choice. IMHO clusters should be the same except currently they are "outside the config" since the choice of the cluster affects the API endpoint the tool has to talk to.

In my ideal scenario my colleagues would just need to know which file to "apply". The file itself (through its name or directory location or comments or more documentation) will guide the user to the meaning of what environment that actually is (dev, staging, production, some well knownv deployment X)