Hacker News new | ask | show | jobs
by solatic 1023 days ago
Super interesting. I can clearly see how this improves the caching story compared to vanilla Kubernetes+containerd, and therefore the potential this has to improve autoscaling performance.

Of course, it understandably requires the host VM to be NixOS, which means my Kubernetes cluster needs to be running NixOS, which means I would need to abandon the ease afforded by EKS/GKE... which is a tall ask. It would be great if there was a way to more easily add this into popular managed Kubernetes distributions, to play around more with running "Nix images", simply by adding nodeSelector and/or node affinity to the Kubernetes manifests.
3 comments
hinshun 1023 days ago
We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service.

I have tested with EKS, and you should be able to use nix-snapshotter with GKE too. May be able to put together some docs for that later.

In the meantime, see this for running a different snapshotter on EKS: https://blog.realvarez.com/using-estargz-to-reduce-container...

link
mikepurvis 1023 days ago
I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time, so for large closures you give up a lot of the benefit of the cluster being able to cache layers (as in, say, a nix2container- or nixery-style image) and achieve very fast subsequent startups.

I'm a k8s novice, but would there be a way to run the nix daemon/cache in a semi-persistent pod on each node, and then "attach" it to the actual worker pods?

link
rgoulter 1023 days ago
> I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time

I got the impression that it uses the node's nix daemon.

From the project readme's FAQ https://github.com/pdtpartners/nix-snapshotter#faq

> What's the difference between this and a nix-in-docker?

> If you run nix inside a container (e.g. nixos/nix or nixpkgs/nix-flake) then you are indeed fetching packages using the Nix store. However, each container will have its own Nix store instead of de-duplicating at the host level.

> nix-snapshotter is intended to live on the host system (sibling to containerd and/or kubelet) so that multiple containers running different images can share the underlying packages from the same Nix store.

link
mikepurvis 1023 days ago
Yeah, I think that's the "main" idea of it, but it was also mentioned that this can work in GKE/EKS if you start a new daemon each time.
link
hinshun 1022 days ago
The EKS/GKE integration involves modifying the host that the kubelet lives and adding nix-snapshotter as a sibling host service.

I didn’t mean running nix-snapshotter as a Kubernetes resource because then there’s a chicken & egg problem. Kubernetes needs nix-snapshotter image service to resolve the nix-snapshotter image.

link
solatic 1023 days ago
I would definitely appreciate seeing some docs with quick pointers on setting that up for EKS at least, if not also GKE!
link
adobrawy 1023 days ago
I'm not sure exactly how the communication of components works there, but in k8s can this be deployed using daemonset rather host service?
link
sporeray 1023 days ago
I don't think this is true, I think it only requires the host system to have nix installed.
link