[vishvananda]

The point of crashcart is to be able to side load a filesystem with utilities into a running container. It is very important that the location we pick doesn't conflict with any path already in the container. If we used /nix as the mount path it would conflict with any container that uses nix. In order to prevent this (probably rare) conflict, we build our utilities in /dev/crashcart/ instead.

[catern]

Hmmmm, interesting... That does seem like a pretty good reason, though maybe you could bind mount over /nix? That should be fine from a Nix perspective, not sure how well it works with container technologies.

[vishvananda]

we could bind mount over /nix or /nix/store, but that means any existing nix packages from the container would not be available. The whole point is to have the whole container file system available along with the utilities. We could alternatively find each package that we need for our debugging utilities and bind mount each directory from the store individually. This would work due to unique paths in the store, but that means potentially hundreds of bind mounts and is an orchestration nightmare.

[catern]

Sorry, I didn't mean bind mount, I mean union mount, like with OverlayFS or whatever the most-used one is.

[vishvananda]

we didn't look at using overlay. Might be possible, although that would introduce a dependency on kernel version and/or module. A custom fuse might be an option here as well but fuse in containers is a bit sketchy at the moment.

[danbst]

Try Nix 2.0, it can place your derivations in custom folder by clever use of chroot, read about --store argument.

[vishvananda]

This looks like a neat feature. Reading up on it briefly, it looks like it changes the mount namespace / chroots when running the binary in order to accomplish this. Unfortunately that defeats the purpose of crashcart, since the whole idea is to have access to the container file system while running your sideloaded binaries.

Unless there is some magic that I'm missing, I don't think this option helps for our use case.

[vishvananda]

Ok. I will check it out. Thanks.