[Helmut10001]

I use Proxmox since 5 years at home. Mostly docker nested in unprivileged LXCs on ZFS for performance reasons. I love the reliability. Proxmox has never lost me. They churn out constant progress without making too much noise. No buzzwords, no advertising, just a good reliable product that keeps getting better.

[dusanh]

Unprivileged LXCs? Interesting, I thought containers would require privileged LXC. At least, that it my takeaway from trying to run Podman in a nesting enabled, but unprivileged LXC under non-root user. I kept running into

> newuidmap: write to uid_map failed: Operation not permitted

I tried googling it, tried some of the solutions, but reached the conclusion that it's happening because the LXC is not privileged.

[LilBytes]

Have to map GUID and UIDs from the Proxmox host to the LxC to allow bind mappings work as an example.

Proxmox doco for unprivileged LxC is here: https://pve.proxmox.com/wiki/Unprivileged_LXC_containers

[dusanh]

Replying to my own comment for posterity. I was able to figure it out!

In the LXC, I created a new non-root user and then set it's uid and git to a LOWER number than what every tutorial about rootless Podman recommends (devel being my non-root user here):

  # cat /etc/subuid /etc/subgid
  devel:10000:5000
  devel:10000:5000

Then I also had to edit configuration for the LXC itself on the Proxmox host to allow tun and have it created on container boot:

   # cat /etc/pve/lxc/{lxc_vmid}.conf
   ...truncated...
   lxc.cgroup2.devices.allow: c 10:200 rwm
   lxc.mount.entry: /dev/net dev/net none bind,create=dir

Note: I have no idea why a lower number of ids works...

[Helmut10001]

Ah.. apologies for my misguidance below. It made me realize that I wrote these blog posts with a VM on another Hypervisor, not on my Proxmox/LXC (just the Docker guide that I haven't yet transitioned to rootless-in-unprivileged-lxc).

See the explanation here [1]: Unprivileged LXC on Proxmox seem to be restricted to the uid range below 65536 UIDs/GIDs (to be used _inside_ the LXC -> to be mapped to > 100000:165536 outside the LXC/on the host).

In order to use subuids/gids > 65536 inside the LXC, add a mapping to the LXC config:

    root:100000:65536

to /etc/subgid and /etc/subuid.

Now you'll have 100000 to 165536 available inside the LXC, where you can add:

    devel:100000:65536

to the /etc/subgid and /etc/subuid inside the LXC, for nested rootless podman/docker.

As a consequence, you're mapping the devel user to the same range as the LXC root user. In other words, processes inside the LXC and inside the rootless podman could run in the same namespace on the Proxmox host. If you don't want that, you'll need to provide additional ranges to the LXC (e.g. root:100000:165536 and then map `devel` to (e.g.) 200000 to 265536 (devel:200000:265536).

* I did not actually test all stated above.

[1] https://forum.proxmox.com/threads/how-to-use-uids-gids-highe...

[Helmut10001]

I once wrote a post about Docker in unprivileged LXC on ZFS [1]. The post is a little bit outdated, as it is much simpler today with ZFS 2.2.0, which is natively supported. There's also a more recent post that shows how to run rootless docker [2], with updated uid-mappings. Both may be helpful, have a look.

The advantage of using LXC for me is resource consumption and separation of concerns. I have about 35 Docker containers spread over 10 LXCs. The average CPU use is 1-3% and I only need about 10GB of memory (even with running bigger containers like Nextcloud, Gitlab, mailcow-dockerized etc.). With docker-compose.yml's, automatic updates are easy and robust.

[1]: https://du.nkel.dev/blog/2021-03-25_proxmox_docker/

[2]: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless...

[dusanh]

Thank you, this made me realize I assigned a wrong (too little) number for uids. It did not fix my issue however. I still see

  (dev) $ podman info
  ERRO[0000] running `/usr/bin/newuidmap 3427 0 1000 1 1 100000 65536`: newuidmap: open of uid_map failed: Permission denied
  Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

I tried a solution I found on Red Hat's Customer Portal:

  (root) # setcap cap_setuid+ep /usr/bin/newuidmap
  (root) # setcap cap_setgid+ep /usr/bin/newgidmap

Also did not work. I can run

  (root) # podman info 

just fine as root. This leads me to believe there are some other problems with my non-root user permissions.

EDIT: It probably makes little sense, to run rootless on top of an already unprivileged LXC. I just wanted to give vscode server it's own non-root user in there. Oh well...

[Helmut10001]

Yes, just start from scratch and provide uid-mappings from the beginning. Looks like those uids were set from before adding the mappings and it is trying to access uids it is not allowed to access.

I used rootless docker in rootless lxc because the Postgres Docker (e.g.) will try to setup a non-root user by default. In a rootless LXC, this means it will try to access very large uids (>100000), which are not available, unless explicitly prepared.

[dusanh]

That actually did not do anything different for me. I did the following:

1) Created a new LXC.

2) As root, I created a new user "devel"

3) For the "devel" user set both subuid and subgid to devel:100000:65536

4) As root, installed podman

5) In another SSH session, logged in as "devel" and ran "podman version"

Same error as before. This is in a Debian 12 LXC running on Proxmox.

[Helmut10001]

I am also using Docker in Debian 12 LXC on Proxmox. I am not sure what has gone wrong here.