[miquong]

For image and layer manipulation, crane is awesome - as is the underlying go-containerregistry library.

It lets you add new layers, or edit any metadata (env vars, labels, entrypoint, etc) in existing images. You can also "flatten" an image with multiple layers into a single layer. Additionally you can "rebase" an image (re-apply your changes onto a new/updated base image). It does all this directly in the registry, so no docker needed (though it's still useful for creating the original image).

https://github.com/google/go-containerregistry/blob/main/cmd...

(updated: better link)

[lyxell]

This is a great recommendation. It is worth noting that unlike Docker, crane is root- and daemonless which makes it work great in Nix (it's called 'crane' in the Nix repository). This allows for Nix to be used to manage dependencies for both building (e.g. Go) as well as packaging and deploying (e.g. gnu tar, crane).

[pbowyer]

Is there any performance benefit to having fewer layers? My understanding is that there's no gain by merging layers as the size of the image remains constant.

[apt-get]

There are some useful cases — for example, if you're taking a rather bloated image as a base and trimming it down with `rm` commands, those will be saved as differential layers, which will not reduce the size of the final image in the slightest. Only merging will actually "register" these deletions.

[fishpen0]

Less performance and more security. Lots of ameteur images use a secret file or inadvertently store a secret to a layer without realizing an rm or other process in another layer doesn't actually eliminate it. If the final step of your build squashes the filesystem flat again you can remove a lot of potentially exposed metadata and secrets stored in intermediate layers

[miquong]

Eventually, once zstd support gets fully supported, and tiny gzip compression windows are not a limitation, then compressing a full layer would almost certainly have a better ratio over several smaller layers

https://github.com/opencontainers/image-spec/issues/803

[pbowyer]

Is it coming? That ticket doesn't fill me with hope, given its age and the disagreements over backwards compatibility.

[yrro]

If you've got a 50 layer image then each time you open a file, I believe the kernel has to look for that file in all 50 layers before it can fail with ENOENT.

[cmckn]

It depends on your OCI engine; but this isn’t the case with containers. Each layer is successively “unpacked” upon a “snapshot”, from which containers are created.

[yrro]

Interesting. Certainly with podman I observe that the layers are combined by overlayfs.

I build this Dockerfile into a container with 'podman build -t test .':

    FROM registry.access.redhat.com/ubi9/ubi-minimal
    RUN touch /file1
    RUN touch /file2
    RUN touch /file3

Then run it with 'podman run --rm -it test':

Then I can inspect the container's storage configuration:

    $ podman inspect -l | jq '.[].GraphDriver'
    {
      "Name": "overlay",
      "Data": {
        "LowerDir": "/home/sam/.local/share/containers/storage/overlay/e0055211bf4b6a839f91759d9bbbb5b93c61f4e2969b25ec9d732428557c230f/diff:/home/sam/.local/share/containers/storage/overlay/efd666ad5dc6b7a92933043ed42a41341b4e25c80e981f6d01ab41927d8f8a89/diff:/home/sam/.local/share/containers/storage/overlay/1c108fb13c6708a93dcf237bea2637344cf65d3c9272693817aa0741b158fd7b/diff:/home/sam/.local/share/containers/storage/overlay/80c0d7946d02825018d47fbf34df054bec5dc433ae20f71ffabd8d3725448837/diff",
        "MergedDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/merged",
        "UpperDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/diff",
        "WorkDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/work"
      }
    }

Note that there are four directories in the LowerDir, and examining their contents reveals that there's one per layer in my 'test' image:

    $ podman inspect -l | jq '.[].GraphDriver.Data.LowerDir' -r | tr : '\n' | xargs -rt -n1 ls
    ls /home/sam/.local/share/containers/storage/overlay/e0055211bf4b6a839f91759d9bbbb5b93c61f4e2969b25ec9d732428557c230f/diff
    file3  run/
    ls /home/sam/.local/share/containers/storage/overlay/efd666ad5dc6b7a92933043ed42a41341b4e25c80e981f6d01ab41927d8f8a89/diff
    file2  run/
    ls /home/sam/.local/share/containers/storage/overlay/1c108fb13c6708a93dcf237bea2637344cf65d3c9272693817aa0741b158fd7b/diff
    file1  run/
    ls /home/sam/.local/share/containers/storage/overlay/80c0d7946d02825018d47fbf34df054bec5dc433ae20f71ffabd8d3725448837/diff
    afs/  bin@  boot/ dev/  etc/  home/ lib@  lib64@  lost+found/ media/  mnt/  opt/  proc/  root/  run/  sbin/ srv/  sys/  tmp/  usr/  var/

Same for a system (non-rootless) container:

    # podman inspect systemd-oxidized | jq '.[].GraphDriver.Data.LowerDir' -r | tr ':' '\n' | xargs -rt -n 1 ls -F
    ls -F /var/lib/containers/storage/overlay/062693a4d8fe24055645ee1c11bbc542507ba3fe101ed4e5b76b6457752d3ac5/diff 
    home/
    ls -F /var/lib/containers/storage/overlay/9d9c6a906aff40160f924c4fec96692a6fc59f27316b8df5ee2616091008a2fe/diff 
    home/
    ls -F /var/lib/containers/storage/overlay/e35e4cf6ce1057979baee991e609f82afb85360560dc8bf36ab069d8e8288f5f/diff 
    home/
    ls -F /var/lib/containers/storage/overlay/da5189fd0e740204d7a30e4b59b012251e8d36a1db488bf47ede61247a948fff/diff 
    home/
    ls -F /var/lib/containers/storage/overlay/702d48c0e8afce7cf5ce049939d8fdb596755e709a3735bbbf4900ec65c45529/diff 
    etc/  home/  var/
    ls -F /var/lib/containers/storage/overlay/877031909d6cf6038e0e0295c34f14932b71903835b3eb13ae6bf23396d65daf/diff 
    etc/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/b90fc2ebe1d7c19e5ee43033b8e8be95210b4e3a0012bb54db327f6d52e89d12/diff 
    tmp/
    ls -F /var/lib/containers/storage/overlay/aa2e131948d26d04f48f7c01c57aa98e2a308e4fc0d707c9532d30fa4ab89f5d/diff 
    root/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/6af6a203c83f697aa35b5007b5678b8c2b052ab482d9d7f7e5b322cd74682441/diff 
    root/  tmp/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/b237f20a22db5d282838c5a7c9d78f7d6e648d9b223a328fa8b8078e9c6b5bbe/diff 
    ls -F /var/lib/containers/storage/overlay/7d1bc5d46561795a4f7058e4098faa345e119617292761933e2ad7f185dbd45b/diff 
    ls -F /var/lib/containers/storage/overlay/88d52756a706920fbcb38cbbb6eaf43ea09be0045361ffe6920185ca7179a1d9/diff 
    tmp/
    ls -F /var/lib/containers/storage/overlay/0d0dedc3e4f8d7f39b9a0c78d1cf380ad1161023a49bd74ab8b51626f6d21f19/diff 
    root/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/d74968407a57d753c2ae232ea26e0e2d7fed30683a67a993b814b71684817eaa/diff 
    root/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/885b215dc2a8304a58a6d35a5668c74d3d80f27dacf56b97a3b8b13c1a423b33/diff 
    root/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/a4edd3819e3655d0ca90ac898a4a2b91ec50c83cbf61862186e10ae1b44cbcc0/diff 
    etc/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/954dcca00292635597e4c5ee9ebc35bf5fcc15a50dfd8f176df7b4c15e695e10/diff 
    bd_build/  etc/  root/ run/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/fb670aa62e0f4f71382ef61a742e8ab1ad5d345b95267b0adc18c10e6c87f13b/diff 
    bd_build/
    ls -F /var/lib/containers/storage/overlay/7f5cbd8cc787c8d628630756bcc7240e6c96b876c2882e6fc980a8b60cdfa274/diff 
    bin@  boot/  dev/  etc/  home/ lib@  lib32@  lib64@  libx32@  media/  mnt/  opt/  proc/  root/  run/  sbin@  srv/  sys/  tmp/ usr/  var/

('podman system info' confirms that .store.graphDriverName == "overlay")

I'd be interested to see what this looks like with docker?

[naikrovek]

That seems ripe for optimization, if true, especially since those layers are all immutable.

[yrro]

A container runtime could optimize for speed by unpacking all those layers one by one into a single lower directory for the container to use; but at the cost of using lots of disk space, since those layers would no longer be shared between different containers.

[dayjaby]

One simple case where the resulting image is bigger than necessary:

``` COPY ./package.deb /tmp/package.deb

RUN dpkg -i /tmp/*.deb && rm -rf /tmp/*.deb ```

This results in two layers, with one layer containing a huge file, thus being part of the final image if you don't do multi-stage builds.

[momothereal]

I'm working on a tool that does the opposite: to split layers into smaller, deterministic deltas.

[mcpherrinm]

If files are overwritten or removed in a lower layer, there can be size savings from that.

[natebc]

some startup performance savings in fewer http requests to fetch the image. small for sure but it's something?

[electroly]

In practice I've found the performance savings often goes the other way--for large (multi-GB) images it's faster to split it up into more layers that it can download in parallel from the registry. It won't parallelize the download of a single layer and in EC2+ECR you won't get particularly good throughput with a single layer.

[whirlwin]

Depends. If you would have to fetch a big layer often because of updates, that's not good. But if what is changing frequently is in a smaller layer, it will be more favorable