[flimsypremise]

I'll be honest, I just don't think this is a great way to do development in any language:

`ko builds images by executing go build on your local machine`

If you've done any sort of work with service or application development on a team, you will no doubt have encountered issues with relying on local machine build dependencies. Enforcing dependency versions across the team is basically impossible even in cases where the entire team is using the same OS and hardware, so you're going to run into problems where users are building the codebase with different versions of tools and dependencies. This is one of the core problems that containerization was intended to solve, so if you are using containerization in development and not building inside the container, you are missing out on one of the major advantages of the paradigm.

[arp242]

Unlike languages that take a lot of stuff from /usr/lib/ system directories, this is not a huge problem in Go, as it doesn't do that. I'm not saying it's never a problem; but it's fairly rare

[ImJasonH]

I do agree with you in general. However, while containers were being invented to solve this problem, Go was also solving this problem.

For the most part, for the most common simple Go applications, if you build the same code with the same version of Go installed, you'll get the same dependencies and the same artifact.

Building Go applications in containers is not necessary in general, and doing so makes it much more complicated to share a build cache. You can of course do it with enough additions to your Dockerfile, but why bother?

If your developer team and CI environment are all using a similar recent Go, they shouldn't have different behavior using ko together.

[flimsypremise]

See now that right there is the issue. _IF_ your developer team and CI environment are all using a similar recent Go. Maybe you can keep this going with a single application and a small team that is actively maintaining that application, but even then I've run into issues with people using different computer with different architectures. Just this past week I had to solve a problem with a Go repo that wasn't building a container caused by the fact that whoever built out the development configuration was using an Intel Mac and certain aspects of the build didn't work on M1/2 Macs. I solved this by containerizing the entire thing, and now the entire setup for repo is a single command. No need to install any dependencies aside from Docker on your machine, no wondering if you're following the instructions correct, just clone the repo and `make run`.

Now extrapolate this out to larger orgs, where months might pass between changes to repos, and commits might be coming from disparate teams. Trying to mandate specific Go versions AND versions of any developer tooling that you might need to run your Go application locally, and you are inviting chaos.

Always build in the container. Local system dependencies are the enemy. Docker configuration for Go is actually extraordinarily simple compared to most languages, and even something like build caching is easy to handle via volumes.

[matrss]

I agree with this being an issue, although certainly less pronounced with go compared to other languages. I don't think containers solve the issue though: none of the mainstream approaches to building containers actually make the builds (and by extension images) inherently reproducible. It just shifts the issue from differing host systems to differing base layers (granted, the issue is less pronounced with those). As long as not every dependency is fully pinned your builds could break any day and arbitrarily between machines, if you don't build at least at the exact same time.

To solve the issue fully you need a more comprehensive approach to packaging. nix or guix can provide that.

Containers are more useful as a software distribution mechanism.

[flimsypremise]

I would say that containers are the foundation of the practice of reproducible builds. They don't solve the problem on their own, but containerization is a core element of the best practice of reproducible builds, along with utilization of lockfiles and infrastructure as configuration. It's certainly a hell of a lot easier to get a containerized application to a reproducible state than one run on a local machine with an unknown architecture and OS.

[matrss]

> I would say that containers are the foundation of the practice of reproducible builds.

If you mean containers as in the isolation features that are utilised by docker et al. to provide their flavor of compartmentalization then yes, those are pretty useful for reproducibility (although not 100% necessary, I think).

If you mean containers as in a bundled linux userland (so, e.g. a docker container running some image) then no, that is entirely orthogonal to reproducibility, as demonstrated by nix and guix which AFAIK use the low-level isolation features for their build sanboxes but do not have anything resembling a container image involved.

[pjmlp]

Containers predate this common GNU/Linux problem and weren't invented to fix it, rather sandbox applications.