Hey, ko maintainer here! I'd love to answer any questions or hear any feedback folks have.
Ko's simplicity comes from focusing on doing exactly one thing well -- it doesn't have to run containers to build code in any language, it just builds minimal Go containers, and that means it can focus on doing that as well as possible.
Another powerful benefit of focusing on Go (IMO) is that ko can be used to transform a Go importpath to a built image reference.
A common simple use: `docker run $(ko build ./cmd/app)`
This is also how Kubernetes YAML templating[1] works, and it's also how the Terraform provider[2] works -- you give it an importpath, and ko transforms it into a built image reference you can pass to the rest of your config. I don't even "use ko" day-to-day most days, I mostly just use terraform. :)
Ko works ok but lacks a lot of configurability, compared to something like jib, that does end up important. There are a few open issues on it but the commit history is basically dependabot, not much action for UX. Better than nothing I guess but it really does give off the feeling of a project spun off from Google to let it die.
Despite that it's probably still the best tool for building containers in Go, but it's not pleasant.
I'll be honest, I just don't think this is a great way to do development in any language:
`ko builds images by executing go build on your local machine`
If you've done any sort of work with service or application development on a team, you will no doubt have encountered issues with relying on local machine build dependencies. Enforcing dependency versions across the team is basically impossible even in cases where the entire team is using the same OS and hardware, so you're going to run into problems where users are building the codebase with different versions of tools and dependencies. This is one of the core problems that containerization was intended to solve, so if you are using containerization in development and not building inside the container, you are missing out on one of the major advantages of the paradigm.
Unlike languages that take a lot of stuff from /usr/lib/ system directories, this is not a huge problem in Go, as it doesn't do that. I'm not saying it's never a problem; but it's fairly rare
I do agree with you in general. However, while containers were being invented to solve this problem, Go was also solving this problem.
For the most part, for the most common simple Go applications, if you build the same code with the same version of Go installed, you'll get the same dependencies and the same artifact.
Building Go applications in containers is not necessary in general, and doing so makes it much more complicated to share a build cache. You can of course do it with enough additions to your Dockerfile, but why bother?
If your developer team and CI environment are all using a similar recent Go, they shouldn't have different behavior using ko together.
See now that right there is the issue. _IF_ your developer team and CI environment are all using a similar recent Go. Maybe you can keep this going with a single application and a small team that is actively maintaining that application, but even then I've run into issues with people using different computer with different architectures. Just this past week I had to solve a problem with a Go repo that wasn't building a container caused by the fact that whoever built out the development configuration was using an Intel Mac and certain aspects of the build didn't work on M1/2 Macs. I solved this by containerizing the entire thing, and now the entire setup for repo is a single command. No need to install any dependencies aside from Docker on your machine, no wondering if you're following the instructions correct, just clone the repo and `make run`.
Now extrapolate this out to larger orgs, where months might pass between changes to repos, and commits might be coming from disparate teams. Trying to mandate specific Go versions AND versions of any developer tooling that you might need to run your Go application locally, and you are inviting chaos.
Always build in the container. Local system dependencies are the enemy. Docker configuration for Go is actually extraordinarily simple compared to most languages, and even something like build caching is easy to handle via volumes.
I agree with this being an issue, although certainly less pronounced with go compared to other languages. I don't think containers solve the issue though: none of the mainstream approaches to building containers actually make the builds (and by extension images) inherently reproducible. It just shifts the issue from differing host systems to differing base layers (granted, the issue is less pronounced with those). As long as not every dependency is fully pinned your builds could break any day and arbitrarily between machines, if you don't build at least at the exact same time.
To solve the issue fully you need a more comprehensive approach to packaging. nix or guix can provide that.
Containers are more useful as a software distribution mechanism.
I would say that containers are the foundation of the practice of reproducible builds. They don't solve the problem on their own, but containerization is a core element of the best practice of reproducible builds, along with utilization of lockfiles and infrastructure as configuration. It's certainly a hell of a lot easier to get a containerized application to a reproducible state than one run on a local machine with an unknown architecture and OS.
> I would say that containers are the foundation of the practice of reproducible builds.
If you mean containers as in the isolation features that are utilised by docker et al. to provide their flavor of compartmentalization then yes, those are pretty useful for reproducibility (although not 100% necessary, I think).
If you mean containers as in a bundled linux userland (so, e.g. a docker container running some image) then no, that is entirely orthogonal to reproducibility, as demonstrated by nix and guix which AFAIK use the low-level isolation features for their build sanboxes but do not have anything resembling a container image involved.
FROM alpine:latest AS base
# Scratch does not have shell so we have to create non-root user in here and later copy it into scratch.
RUN adduser -D -u 123456 appusr
FROM scratch
# Copy the binary.
COPY foo.bin /foo.bin
# Copy the user and set is as active.
COPY --from=base /etc/passwd /etc/passwd
USER appusr
# Non-root user cannot bind to "privileged" ports(<=1024).
EXPOSE 1234
ENTRYPOINT ["/foo.bin"]
Simple. But i can see ko being good alternative if you for some reason do not want to install docker on your computer but still be able to build docker containers.
Or, if you don't want to install it in your CI/CD. Which could be extremely valuable; some CI/CD providers make "building a docker image" a weirdly hard (and sometimes, even much more expensive) step due to docker-in-docker madness.
In my experience, not using docker to build docker images is a good idea. E.g. buildah[0] with chroot isolation can build images in a GitLab pipeline, where docker would fail. It can still use the same Dockerfile though.
If you want to get rid of your Dockerfiles anyway, nix can also build docker images[1] with all the added benefits of nix (reproducibility, efficient building and caching, automatic layering, etc.).
> But i can see ko being good alternative if you for some reason do not want to install docker on your computer but still be able to build docker containers.
Docker is a much bigger dependency than ko, involving a daemon and a socket you have to manage and secure. ko is a only a go program, and builds are straight-forward and lightning fast compared to Docker.
Importantly, ko also removes the need for a Dockerfile. Yes, as you point out a Dockerfile can be simple, but but the best part is no part. Dockerfile has plenty of foot guns and best practices to learn so if at the end of the day all you need is a go binary on a container ko will serve you much, much better.
That works, but it also means maintaining a Dockerfile that does the COPYs, when all you're really doing is assembling image layers. This is more or less exactly what ko does, without using the Dockerfile language to describe it.
Not having to run a Docker daemon is just a nice bonus! :)
This is not talking so much about the Docker GUI things but rather this doesn't require a Docker daemon at all.
Similar tools are Google's Jib for JVM based applications, `rules_docker` and `rules_oci` for Bazel etc.
All of which allow you to build applications without needing a Docker runtime at all during the build process which is a huge plus for portability, performance and repeatability.
From my experience, podman desktop is pretty good at containers, but horrible for Kubernetes (it uses KIND but you can't configure basic things, like exposing NodePort IIRC).
Given that SUSE uses Podman, I hope they'll modify Rancher Desktop to use Podman, in which case it'd supersede Podman Desktop.
I think Bazel can be a good fit for larger polyglot organizations that need to manage large codebases in many languages in a uniform way. Basically Google-circa-2010-sized organizations, coincidentally!
For smaller teams, adopting Bazel too early can be a real productivity drain, where you get all of the downsides of its constraints without as many of its benefits. Bazel is overkill for a project of ~10 Go apps, for example. Ko was actually created to help such a project (Knative) migrate off of Bazel's rules_docker to something better, and I think it achieved the goal!
If you have your project packaged with nix (which shouldn't be too complicated if your build process is not too special) it is trivial to turn a nix derivation into a docker image: https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-dockerToo.... As a bonus, it is completely language-agnostic.
Dammit, why didn't I know about this?
In general, I always have the feeling I'm not aware of great applications out there, and I confirm this sensation every time I find them, like now!
How do you all stay tune of the great apps out there?
I track GitHub's trending page for Go, Rust, and occasionally Python projects. Limiting to these languages tends to yield the kinds of projects I'm interested in, just by the nature of the demographic that uses them. Sometimes I browse recent repo activity on Sourcehut too.
Ko's simplicity comes from focusing on doing exactly one thing well -- it doesn't have to run containers to build code in any language, it just builds minimal Go containers, and that means it can focus on doing that as well as possible.
Another powerful benefit of focusing on Go (IMO) is that ko can be used to transform a Go importpath to a built image reference.
A common simple use: `docker run $(ko build ./cmd/app)`
This is also how Kubernetes YAML templating[1] works, and it's also how the Terraform provider[2] works -- you give it an importpath, and ko transforms it into a built image reference you can pass to the rest of your config. I don't even "use ko" day-to-day most days, I mostly just use terraform. :)
1: https://ko.build/features/k8s/ 2: https://ko.build/advanced/terraform/