[tekeous]

Podman was good when it supported systemd unit files, so I could auto start and auto update containers, even entire pods with systemd.

Then they removed that in favor of Quadlet. Now in order to do a single container I can do a unit file, but for a pod, I need to use a Kubernetes cluster definition.

Plus, unlike Docker their containers bow to SELinux definitions, so I have repeatedly struggled with containers unable to access mapped directories.

So what is it, Podman? Should I just use Kubernetes? Should I just make dedicated directories for everything instead of mapping logical places for things?

[lolinder]

I already was defining my infrastructure with docker-compose.yml files, and found out that podman-compose has a poorly documented feature that generates systemd units. It doesn't use the now-deprecated podman feature, it writes the unit files itself, and I find the process much smoother than the podman feature anyway.

To enable the feature:

    $ podman-compose systemd -a create-unit

To register a systemd unit:

    $ podman-compose systemd -a register
    $ systemctl --user enable --now "podman-compose@$PROJECT_NAME"

Updating looks like this:

    $ podman-compose pull
    $ systemctl --user restart "podman-compose@$PROJECT_NAME"

($PROJECT_NAME is usually the directory name.)

Source code for the feature is here if you care to inspect it:

https://github.com/containers/podman-compose/blob/f6dbce3618...

Admittedly, I'm still on podman 4.3.1, but I don't see any reason why this would stop working in later versions of podman.

[ape4]

Its documented. If you just type `podman-compose` in the command line you get...

    usage: podman-compose [-h] [-v] [--in-pod in_pod] [--pod-args pod_args]
                          [--env-file env_file] [-f file] [-p PROJECT_NAME]
                          [--podman-path PODMAN_PATH] [--podman-args args]
                          [--podman-pull-args args] [--podman-push-args args]
                          [--podman-build-args args] [--podman-inspect-args args]
                          [--podman-run-args args] [--podman-start-args args]
                          [--podman-stop-args args] [--podman-rm-args args]
                          [--podman-volume-args args] [--no-ansi] [--no-cleanup]
                          [--dry-run]
                          {help,version,wait,systemd,pull,push,build,up,down,ps,run,exec,start,stop,restart,logs,config,port,pause,unpause,kill}
                          ...
    
    command:
      {help,version,wait,systemd,pull,push,build,up,down,ps,run,exec,start,stop,restart,logs,config,port,pause,unpause,kill}
     ...
        systemd             create systemd unit file and register its compose stacks
                            
                                When first installed type `sudo podman-compose systemd -a create-unit`
                                later you can add a compose stack by running `podman-compose systemd -a register`
                                then you can start/stop your stack with `systemctl --user start podman-compose@<PROJ>`

[lolinder]

Yeah, I eventually found that, but only after I finally stumbled upon someone referencing the feature in a GitHub issue.

The --help is fine documentation for the people who have already installed the tool, but it doesn't help people like OP who just want a simple way to run multiple containers as a systemd unit and don't yet know that podman-compose has a solution.

That's why I said "poorly documented" instead of "undocumented". It's there once you know where to look.

[tw04]

That’s… not documentation. That’s a CLI helpfile. It’s better than nothing but also what is completely broken with the “move fast and break things” mindset.

[scrozart]

Sure looks like documentation to me, but I don't need pretty webpages. 'man' is more than sufficient for tools like these.

[ClumsyPilot]

How would I ever read this documentation without installing postman? Am I suppose to download and install every tool just to read about how it works?

[notatoad]

it's documentation enough to not call it an undocumented feature.

the concern with truly undocumented functionality is that it's not included intentionally: either it's a bug or an experimental feature that could be removed or changed with no notice. a poorly-documented feature, on the other hand, will probably at least get a deprecation notice before it disappears

[lolinder]

> it's documentation enough to not call it an undocumented feature.

Which is why I didn't call it that, I called it poorly documented.

[kccqzy]

That's much more elegant than my hacky hand-written systemd unit (one-shot, remain after exit) to start podman containers.

[notso411]

I thought docker compose was for local dev only and not meant to be used for production workloads?

[lolinder]

I'm using podman-compose for my homelab, which is obviously fine.

But even for small-scale single-node production use cases, I suspect that podman-compose with systemd doesn't have the same concerns as docker-compose does. Since you're registering the workload with systemd, it'll restart with the node as easily as any other service, and rootless containers are a big win for security.

Where you can't keep using (podman|docker)-compose is when you have to scale up a service beyond a single node.

[morelisp]

You can keep using x-compose on several nodes, you just need e.g. ansible or salt on top of it. For many things this is still a local maximum compared to a K8s cluster or "just ssh in'.

[jmnicolas]

Why not docker swarm then?

[morelisp]

For a lot of the services I'm thinking of for this case the scheduling may be across multiple nodes but it may not be flexible - e.g. this is my preferred way to run things which need guaranteed local iops. So Swarm maybe helps with service discovery etc. but its main purpose is lost.

I have a lot of complaints about Docker Swarm but they're either about its ownership issues or relatively minor (but - lots of minor) issues, if you want to use it it's fine. But you do still need an orchestration layer above it anyway.

[lmeyerov]

It's great in prod. Doesn't make $ for companies marketing k8s iaac devops, so no one to advertise.

[dboreham]

Why?

[bonzini]

> unlike Docker their containers bow to SELinux definitions,

That's a bug in docker. If your system isn't configured for SELinux, disable it.

Also the systemd files generated by podman-generate-systemd are just executing "podman start containername", you can write them on your own easily but (unlike e.g. docker-composr) the containers are black boxes pretty much.

The advantage of quadlet is that the definition of the container is declared in the .container file; before I used to write the podman run command line manually in a handwritten systemd unit, and quadlet is a big improvement in that respect and can be an alternative to docker-compose (with advantages and disadvantages).

[INTPenis]

>That's a bug in docker. If your system isn't configured for SELinux, disable it.

I feel obligated to say that you should set it to permissive mode, never disable SElinux.

    sudo setenforce 0

[bravetraveler]

I know it's a hard sell, but the 30 minutes to an hour to understand filesystem contexts and how to copy them from a good place isn't a lot

It's likely to little gain. I know why people don't, but it is very accessible to those open to it

The more one adheres to the FHS, the easier SELinux is

[sph]

I have been doing Linux sysadmin for 20 years and I just stopped trying to understand SELinux. It looks and feels like an abomination borne out of some IBM or other antediluvian corporate UNIX system for programmers wearing suit and tie.

Terrible documentation, terrible mental model, terrible CLI UX, terrible error messages.

I run Fedora and SELinux is working well enough, but it's a piece of machinery I can't wait to see replaced, however useful people swear it is.

[Anarch157a]

It was originaly developed by the NSA, so it's even worse than corpos wearing suit and tie, it came from spooks.

[voidfunc]

Probably intentionally complex for just that reason. Easy to leave a misconfiguration.

[bravetraveler]

I totally get it. I exert next to zero effort to keep it around for my systems!

For systems that shouldn't do much more than exactly what is prescribed, it's acceptable, is what I'm after... I guess.

I can't do justice to the source, but there's a concept about our programs/creations reflecting us.

Like a peer hints - SELinux reflects an agency like the NSA, draconian. Good and bad

[Modified3019]

>understand filesystem contexts and how to copy them from a good place

I’m not really sure what this means (I have 0 knowledge on selinux)

[bravetraveler]

edit: Apologies for the wall. I think I finally landed on a decent mix after many edits. I'm finished now, lol.

SELinux has a bit of a well deserved reputation... but I, a fairly silly person, have managed to work with it

This video likely explains things far better than I can in this post:

https://www.youtube.com/watch?v=_WOKRaM-HI4

I'll probably fail with specifics, where they certainly do a better job.

So. First it's important to know SELinux runs in one of two modes:

    * A targeted mode where well-known/accounted-for things are protected. For example, nginx
    * A more draconian mode where *everything* is protected

People often present the first [default] mode as if it were the second.

The protection is based on policies that say 'things with this label/at this path are allowed to do XYZ'.

It's very focused on filesystem paths and what relevant applications try to do.

It's entirely manageable, but admittedly, complicated. Without practicing the words I can't express them.

Most people having trouble with SELinux are defying some convention. For example: placing application scratch data in '/etc'.

Policy management is a complicated topic.

The policy can be amended in cases where the standard doesn't apply; I won't cast judgement - sometimes it's a good idea, sometimes not.

Another way to handle this is to copy the label from one path and apply it to the one your application requires/customizes. This is less durable than leaning on the policy.

It acts as a sort of central DB... the goal is to make things such that the policy stores all of the contexts so the files/dirs can have "labels" applied for SELinux

[Modified3019]

Very enlightening, I really appreciate the time you took to outline all of this, thank you!

[bravetraveler]

Some secondary reading, I've referred to FHS at times. It's the 'Filesystem Hierarchy Standard':

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html

They provide guidance on how a given filesystem path should be used.

This has informed the default SELinux policies greatly; familiarity turns hassle into informed assumptions/ease.

[nineteen999]

Only if you want it back in enforcing mode after the next reboot. If you want to make the change permanent, you need to set the following in /etc/selinux/config as well:

SELINUX=permissive

[Kwpolska]

You should set it to permissive mode if you think you will ever need to re-enable that trainwreck.

[bonzini]

I meant disable labeling in podman. :)

[bravetraveler]

Long time fan, yet - agreed.

I keep forgetting this transition happened until I try to 'podman generate systemd [...]'

This is rare because I wrote an Ansible role to do this in a way that feels nice.

Anyway, it really feels like podman lost the mark. I've already subscribed to the unit file maintenance/relationship planning thing. Just let me use the generator. I don't care about Quadlets or how they might be better.

[emerongi]

> Plus, unlike Docker their containers bow to SELinux definitions, so I have repeatedly struggled with containers unable to access mapped directories.

Add the following to containers.conf if you don't want to deal with it:

   [containers]
   label=false

If you don't like podman's default security level, there is usually a way to turn things off.

[sph]

The only times I had issues with SELinux and podman, it was because I forgot to add the z flag to the volume:

    podman run -v .:/app:z image

This only happens locally as files in your home have strict security rules, never had any issue on a CentOS server.

[PhilippGille]

Note there's also uppercase Z, for when the volume shouldn't be shared with other containers: https://unix.stackexchange.com/questions/651198/podman-volum...

[Cyph0n]

I recently migrated over to NixOS which treats systemd as the source of truth for everything, including containers. I found this model extremely intuitive, but it was difficult to apply this to Docker Compose without a lot of manual migration. So I ended up writing a tool that handles this for you — it converts your Compose files into a NixOS config that can be interpreted and managed natively.

https://github.com/aksiksi/compose2nix

[jay-aye-see-key]

This is a cool idea. Two questions:

Arion can wrap docker-compose and run as a project or part of a nixos config. Did you come across Arion before creating this, and have you compared them?

I had a brief look through your examples and it doesn’t look like compose2nix implements docker-compose’s network per compose file. Is this something you want to add?

[Cyph0n]

So, from what I understand, arion provides a Nix frontend for Docker Compose. This allows you to write Nix that runs via Docker Compose. It doesn’t solve the migration problem: if you have an existing Docker Compose project, you still need to manually convert it into Nix for arion to consume.

My tool does the opposite: it takes a Compose file and converts it into OCI containers in Nix. The idea is that your Compose file is the source of truth, and you simply generate Nix to run on NixOS. One benefit here is that you can easily migrate an existing Compose project into native Docker/Podman containers running on NixOS. This removes Docker Compose from the equation entirely - essentially a “reimplementation” of Compose.

It should support the default network per Compose project. See: https://github.com/aksiksi/compose2nix/blob/6dc451fd960f7a9b.... If you’re talking about something else, please feel free to open an issue and I can take a look.

[BossingAround]

> Plus, unlike Docker their containers bow to SELinux definitions

Isn't that great? You want SELinux if you deploy containers on prod.

> I have repeatedly struggled with containers unable to access mapped directories.

Use the `z` or `Z` flag, e.g. `podman run -v mydir:podman_dir:z nginx` to get SELinux labels updated automatically (and temporarily) by podman.

[freedomben]

That was a cool feature (I didn't realize it was gone, that's unfortunate), although I felt the generated code wasn't super great, and if the container is stateless (excepting what's stored in volumes of course) then it's so simple to write your own systemd unit file that I just do that now. I wrote it once and pretty much just copy/paste it when needed, changing the podman run command for image names, port numbers, volumes, etc. For example, here's what I use for Jellyfin. Just drop at `/etc/systemd/system/jellyfin.service`:

    [Unit]
    Description=Jellfyin
    After=NetworkManager-wait-online.service
    
    [Service]
    Restart=always
    ExecStart=/usr/bin/podman run --rm --volume '/srv/jellyfin/config:/config:Z' --volume '/srv/jellyfin/cache:/cache:Z' --volume '/media/media:/media:Z' --publish '8096:8096' --publish '8920:8920' --publish '1901:1900/udp' --publish '7359:7359/udp' --name 'jellyfin' 'docker.io/jellyfin/jellyfin:latest'    
    ExecStop=/usr/bin/podman stop -t 5 jellyfin ; /usr/bin/podman rm -f jellyfin
    
    [Install]
    WantedBy=local.target

Note: You can also just `s/podman/docker/g` and reuse the same service file with docker, which is really convenient for systems where you have no choice

[rcarmo]

This. Quadlet wasn't really needed and just complicated matters for me, so I went back to docker-compose.

[hinkley]

I’ve been complaining about our OPs people pushing Colima over podman and now you’re making me think my podman info is outdated. Ouch.

[madspindel]

You can generate Kubernetes yaml files with podman kube generate. Then you can use systemd to run the generated files: https://www.redhat.com/sysadmin/kubernetes-workloads-podman-...

[runiq]

AFAIK podman either already supports pods in quadlet container files, or will in the near future. https://github.com/containers/podman/pull/20762

[starttoaster]

> So what is it, Podman? Should I just use Kubernetes?

If you're talking about a production system for any business larger than a 10 person tech startup; yeah, probably. Alternatively there's Docker Swarm and Hashicorp Nomad. Though Swarm is not nearly as flexible, it's just easy to use. And Nomad... well, let's just say I've been paying closer attention to Hashicorp's build processes in their open source repos like Packer and Vault as of late and they do some stuff that seems shady to me so use at your own risk.

[_cenw]

`podman generate systemd` is still there, and I see no reason you couldn't use it. it's just a bunch of podman commands wrapped in a unit file, no magic.

feels like a lot more cruft than quadlets to me though.

[cpuguy83]

Docker does use selinux policies.

Mapping directories from the host requires that you change selinux labels on those files so that the container process can access the files. That's just how selinux works.

[INTPenis]

What are you talking about? Quadlet generates ordinary systemd service units with podman commands. Isn't that what you were using?

It's just a simplified layer ontop of those old systemd units.