[hinshun]

Hi flurie, Nixery exposes an API (in the form of an Docker registry) to dynamically build Nix-based images, but still tar & compresses Nix packages into layer tarballs. Since the image spec has a limit of 128 layers (due to overlayfs), a heuristic is used to put popular packages together. However, in practice there is still a large amount of duplication between images that share the same packages due to this heuristic based strategy. You also may deploy the same packages outside of containers but have to duplicate that same data (in a slightly different format) on a Docker Registry.

With `pkgs.nix-snapshotter.buildImage`, containerd natively understands Nix packages, so everything is pulled at package granularity without the layer limit. This means all container content is either already in your host nix store or fetched from your Nix binary cache. Think delta image pulls, which isn't possible with regular images since one bit difference will change the layer hash and thus duplicate.

That said, nothing stops Nixery from building nix-snapshotter images. Then we'll have an Docker Registry that dynamically builds native Nix images.

[0x457]

So if the package isn't in cache, then suddenly k8s node is building it from source?

[hinshun]

If you are pushing nix-snapshotter images to a Docker Registry, then it can only either use what's in your host nix store, or fetch from a binary cache.

The same is true if you are using the special image reference like `nix:0/nix/store/f8b1hia3hcqwa5d46anzy3cszi3s6ybk-nix-image-redis.tar`, it just means that instead of fetching the image manifest from a Registry, you are using the Nix protocols.

You will only build from source dynamically (and a mix of caching) if you are doing `kubectl apply -f ${podSpec}` as a full deployment Nix expression. See the README's asciinema and this file as an example: https://github.com/pdtpartners/nix-snapshotter/blob/v0.1.0/e...

[alex_hirner]

I think your usage example will be easier to understand, if it explains where nix:0 is setup as CRI image-service. Maybe I glanced over it.

Awesome project!

[hinshun]

Hi Alex, I’ve updated the README with a bit more information.

0 is actually an unbindable port, so nix:0 is just an arbitrary string that is unlikely to have conflicts. The Kubelet needs to be configured with —image-service-endpoint to use the nix-snapshotter gRPC socket to handle the “PullImage” RPC. There we can take over resolving the image reference when it’s prefixed with nix:0.

[0x457]

Oh, I see. Thank you for explaining. I misunderstood how this works.