[Ramiro]

Distroless are tiny, but sometimes the fact that don't have anything on them other than the application binary makes them harder to interact with, specially when troubleshooting or profiling. We recently moved a lot of our stuff back to vanilla debian for this reason. We figured that the extra 100MB wouldn't make that big of a difference when pulling for our Kubernetes clusters. YMMV.

[podge]

I found this to be an issue as well, but there are a few ways around this for when you need to debug something. The most useful approach I found was to launch a new container from a standard image (like Ubuntu) which shares the same process namespace, for example:

docker run --rm -it --pid=container:distroless-app ubuntu:20.04

You can then see processes in the 'distroless-app' container from the new container, and then you can install as many debugging tools as you like without affecting the original container.

Alternatively distroless have debug images you could use as a base instead which are probably still smaller than many other base images:

https://github.com/GoogleContainerTools/distroless#debug-ima...

[jrockway]

I've found myself exec-ing into containers a lot less often recently. Kubernetes has ephemeral containers for debugging. This is of limited use to me; the problem is usually lower level (container engine or networking malfunctioning) or higher level (app is broke, and there is no command "fix-app" included in Debian). For the problems that are lower level, it's simplest to resolve by just ssh-ing to the node (great for a targeted tcpdump). For the problems that are higher level, it's easier to just integrate things into your app (I would die without net/http/pprof in Go apps, for example).

I was an early adopter of distroless, though, so I'm probably just used to not having a shell in the container. If you use it everyday I'm sure it must be helpful in some way. My philosophy is as soon as you start having a shell on your cattle, it becomes a pet, though. Easy to leave one-off fixes around that are auto-reverted when you reschedule your deployment or whatever. This has never happened to me but I do worry about it. I'd also say that if you are uncomfortable about how "exec" lets people do anything in a container, you'd probably be even more uncomfortable giving them root on the node itself. And of course it's very easy to break things at that level as well.

[theptip]

Also if you are running k8s, and use the same base image for your app containers, you amortize this cost as you only need to pull the base layers once per node. So in practice you won’t pull that 100mb many times.

(This benefit compounds the more frequently you rebuild your app containers.)

[yjftsjthsd-h]

Doesn't that only work if you used the exact same base? If I build 2 images from debian:11 but one of them used debian:11 last month and one uses debian:11 today, I thought they end up not sharing a base layer because they're resolving debian:11 to different hashes and actually using the base image by exact image ID.

[gui77aume]

Indeed. But the old debian and the new debian images may have a common layer.

[PaulKeeble]

Base images like alpine/debian/ubuntu get used by a lot of third party containers too so if you have multiple containers running on the same device they may in practice be very small until the base image gets an upgrade.

[erik_seaberg]

This. The article talks about

> Each layer in your image might have a leaner version that is sufficient for your needs.

when reusing a huge layer is cheaper than choosing a small layer that is not reused.

[Ramiro]

I think this something that people miss a lot when trying to optimize their Docker builds. Is the whole optimizing for most of your builds vs optimizing for a specific build. Not easy.

[gravypod]

There are some tools that allow you to copy debug tools into a container when needed. I think all that needs to be I'm the container is tar and it runs `kubectl exec ... tar` in the container. This allows you to get in when needed but still keep your production attack surface low.

Either way as long as all your containers share the same base layer it doesn't really matter since they will be deduplicate.

[theptip]

I believe “Ephemeral containers” are intended to resolve this issue; you can attach a “debug container” to your pod with a shell and other tools.

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral...

Still beta, I haven’t tried it yet myself. Looks interesting though.

[staticassertion]

The way I imagine this is best solved is by keeping a compressed set of tools on your host and then mounting those tools into a volume for your container.

So if you have N containers on a host you only end up with one set of tooling across all of them, and it's compressed until you need it.

You can decouple your test tooling from your images/containers, which has a number of benefits. One that's perhaps understated is reducing attacker capabilities in the container.

With log4j some of the payloads were essentially just calling out to various binaries on Linux. If you don't have those they die instantly.