[noodlesUK]

I think many of these issues are also solved by Podman and systemd depending on what kind of "production" you're building for. If you're building a linux-y appliance and you need to run a few containers I think Podman is a much better and more ergonomic way of doing so. I think perhaps that's less true for running a web service (where the linux environment is just a means to that end).

[pmig]

What are the benefits of running Podman Compose instead of Docker Compose? I don't see how it helps with orphan containers, logs and mutable tags.

[whilenot-dev]

GP is talking about podman with generated systemd unit files (a.k.a. podman quadlet[0]), not the docker-compose-compatible podman-compose ...and I'd agree, systemd can manage services on a system just fine, and even better than any compose workload ever could.

journald will help with logs, and the pull policy[1] helps with mutable tags. What help do you need with "orphan containers"?

[0]: https://docs.podman.io/en/latest/markdown/podman-quadlet.1.h...

[1]: https://docs.podman.io/en/latest/markdown/podman-image.unit....

[figmert]

I'm not OP, but the whole podman compose topic gets quite confusing, as initially Podman didn't seem to know what they were trying to do. I've given some more context around it in previous comments.

You shouldn't be using podman compose. It's flimsy and doesn't work very well (at least it was last time I used it prior to Podman v3), and I'm pretty sure it doesn't have Red Hat's direct support.

Instead, activate Podman's Docker API compatibility socket, and simply set your `DOCKER_HOST` env var to that socket, and from there you can use your general docker client commands such as `docker`, `docker compose` and anything else that uses the Docker API. There are very few things that don't work with this, and the few things that don't are advanced setups.

For what it's worth, podman has also a thin wrapper (podman compose) which executes `docker-compose` or the old `podman-compose`. The docs should explain which it picks.

Note:

- `podman-compose` is an early attempt at remaking `docker-compose` v1 but for Podman. This used parsed the compose config and converts them to podman commands, and executes it.

- Later Podman wrote a Docker compatible socket instead, which can work with most docker clis that accept a `DOCKER_HOST` argument, including `docker` and `docker-compose` (both v1 and v2)

- `podman compose` is a thin wrapper that automatically selects `docker-compose` or `podman-compose` depending on which is installed.

Generally all you need is podman, docker-compose (the v2 binary), and that's it. From there you can use `podman` and/or `podman compose`.

[pydry]

One of the nastiest aspects of migrating from docker to podman really is "what to do about docker compose?" coz there are three wildly divergent ways to answer that all of which really suck under certain specific circumstances.

Im no fan of docker and podman by itself is a step up but orchestration headaches are enough to ruin that.

[figmert]

> "what to do about docker compose?"

I don't understand what you're asking here. The answer to that is probably nothing. That is unless you want:

- systemd to manage your containers - You want to use K8s primitives (which are mostly compatible)

I'm unsure what the 3rd method is you're talking about. The nice thing about Podman's compose API is you don't have to change anything (mostly). You can point all your docker tooling to Podman's socket, and it'll (mostly) magically work.

[pydry]

the three options are:

* use systemd, red hat's favorite kitchen sink for handling everything from setting up sound services to mounting your home dir to logging so why not this too i guess.

* docker compose where i have to run a whole separate podman service to lie to docker compose about not actually being docker.

* podman compose which would be the obvious solution if it didnt just plain suck.

[stryan]

> * use systemd, red hat's favorite kitchen sink for handling everything

Systemd is a tool for managing services. Containers are services. Why require an entirely separate bespoke service manager when you're already running one?

> * docker compose where i have to run a whole separate podman service to lie to docker compose about not actually being docker.

This is the same system state as using docker compose with docker: you have a client program speaking to a backing daemon. Only difference here is the Podman service, being daemonless, only runs when needed (assuming you're setting up things the documented way by enabling the podman socket).

> * podman compose which would be the obvious solution if it didnt just plain suck.

Yeah I haven't had the best luck with it either. But part of the reason it's languished is that it makes more sense to just reimplement the Compose spec on the backend rather than re-invent the wheel and create a new compose client as well.

There's also the fourth option of writing Kubernetes yaml and applying that with `podman kube play`. Honestly this is probably closer to being the podman equivalent of docker compose but since it involves writing The Bad YAML (kubernetes) rather than The Good YAML (compose) most people don't use it.

[PxldLtd]

This is what stopped me from picking up Podman more, all our devs use Docker and have been writing compose files for years now. When the response at the time was "you're using Podman wrong, Quadlets are the hot stuff now" it just felt like too big a risk and commitment to jump to at the time. Have things settled more? Getting away from Docker is a bigger priority nowadays for us.

[pydry]

podman compose has shifted from "basically never works" to "if an existing YAML isnt too complex it works" but it's not a drop in replacement yet.

i also want to stay the hell away from quadlets or any other software which tries to make me use systemd more.

[greenicon]

Docker (Compose) has some quirks compared to Podman (Compose), e.g. when using gvisor or a lot of internal networks. Depending on what you do your milage will vary, though.

[kitd]

Agreed. I found compose overlay files merged list values differently between the Docker and Podman versions, which was a PITA in teams running Docker & Linux dev machines.

[cyberax]

FWIW, most of these issues were recently fixed in podman-compose. I can now use the current git version of podman-compose interchangeably with docker-compose.

And one nice thing about podman-compose is that it's ONE PYTHON FILE. You can just copy it into your source tree.

[cyberax]

Podman in general is much higher quality than Docker. For the last 5 years, Docker was throwing all kinds of stuff against the wall to see what sticks, neglecting foundational issues.

Moreover, podman-compose integrates really easily with systemd. You can create a service by just running "podman-compose systemd" and following the prompt! No quadlet nonsense required.

[gear54rus]

Then you learn podman can't even list containers for all users properly and it kind of starts smelling like the whole ip4 vs ip6 debacle: bunch of vocal proponents wanting you to subject yourself to endless torture for no discernible reason.

[figmert]

What do you mean it can't list containers for all users?

[tovej]

I mean, ipv6 is for not runnig out of IP addresses. There is a clear discernible reason.

There are workarounds to make ipv4 work, but they complicate the system and make it more fragile.

[madspindel]

Yes, I recommend this: https://www.redhat.com/en/blog/kubernetes-workloads-podman-s...

[philipallstar]

Is there a nice guide for podman that includes quadlets (or saying not to use them?) I find lots of guides stray into things that work on redhat, and on my Linuxes of choice, Raspbian and Ubuntu, things aren't straightforward.

[notme43]

I find the podman man pages quite readable and thorough if you've had experience configuring systemd services. Good examples as well.

https://docs.podman.io/en/latest/markdown/podman-systemd.uni...

[philipallstar]

Thanks - you're right. I just have never known whether to quadlet or not to quadlet, as the centre of gravity seems to be moving that way, but that might be a redhat-first feature.

[MrDrMcCoy]

The quadlet feature is just part of podman, which consists of no more than a generator for systemd. Generators are just an executable "hook" that spits out unit files according to systemd-defined paths. Any systemd system with podman will be equivalent to any other for the respective versions of each. There is nothing distro-specific about it, especially since podman can be a single multicall binary like busybox to provide it to a system that has systemd and nothing else.

[exceptione]

Can't comment on Raspbian, but Ubuntu LTS (has/had) a seriously outdated podman version. This is the kind of nuisance the Debian derivatives have been running into for more than 20 years: they are extremely conservative, and if that is all you need, then that is great, but if not, you'll have to either run the latest Ubuntu (not LTS), or you upgrade to something like fedora.

[skydhash]

> they are extremely conservative, and if that is all you need, then that is great

You don’t need to live at the edge of new features. Do you upgrade your fridge and your oven every two months? It’s nice when you can have something running and not worry that the next update will break your software and/or your workflow.

[exceptione]

Sure, but these are development dependencies we are talking about. Running old versions of these dependencies block your projects. But it isn't limited to self-developed software, quite often for of the shelve software you run at the same problem.

To each their own, but this is the reason I advice newcomers to stay away from Debian based distro's. I don't intend a distro flamewar, it works perfect for `boring old and feature complete software´ like Dovecot.

To add: containers would alleviate a good part of these concerns, but the stupid thing here is that precisely that is broken for up-to-date podman workflows.

[skydhash]

Your test system should reflect your prod system. Why run Debian if you intend to deploy on the latest ubuntu? Unless you want to use VMs. For other stuff that does not alter the system that much, you can find more recent version in the backports.

[exceptione]

It has integration with systemd, but moreover, I think the promise of Debian-derivatives is one of "we are boring and old, but also boringly stable". Now, throwing in backports undermines that promise. I think one is better of with a distro that moves faster.

[linsomniac]

Not sure if you consider 5.7.0 (6 months old) "seriously outdated", or are talking about Ubuntu 24.04 (the previous LTS). I recently looked and decided 5.8.2 (3 weeks old), didn't have anything compelling to make me want to try to shoehorn it in.

[exceptione]

Ubuntu 24.04. The new LTS had dropped only two weeks ago. LTS users had a very outdated podman (4.9, two years old) and couldn't use quadlet types like build units (v5.2.0, aug 2024).

[linsomniac]

We are switching our Docker systems over to using Podman, primarily to get rid of the machinations we have to do to keep "apt update" from taking down services if there's a new Docker version. We're rolling them up from 24.04 to 26.04 and just using the podman packages on 26.04.

[exceptione]

I see, at least the good thing with 26.04 is that you are set for a while.

[mr_mitm]

In many cases, Debian unstable is also a good choice.

[jiggunjer]

Is there no upstream package repo like docker has.

[MrDrMcCoy]

So use a static build of podman. They're readily available.

[exceptione]

Podman's quadlets have a deep integration with systemd. I guess that if you have that kind of risk appetite you would be better of with running Arch on auto-update.

[MrDrMcCoy]

If by "deep integration" you mean "unit generator", then sure. There isn't much I've seen beyond that, nor are there many systemd features being used that version drift would cause an issue for. Static podman introduces no real risk that I can identify.

[bityard]

I desperately WANT to like podman quadlets and keep trying to find a use case for them. But I always got the impression that the developers who implemented quadlets never actually had to manage multiple containers in a real production environment.

Having your whole application with its containers, volumes, and networks all defined together in one easy-to-read YAML file is a way better experience. Deployment is two steps: 1. `git clone foo` 2. `docker compose up -d`. You can see the state of the application containers with `docker compose ps`. You can run multiple compose applications on the same host and manage them separately by putting them in different directories.

With quadlets, you delegate everything to systemd. You have to break the configuration up into a bunch of tiny unit files and then separately copy them to /etc or a dedicated user's dotfiles. An application with a handful of containers and multiple networks/volumes/etc can spiral into a dozen unit files. Good luck SSH'ing into an unfamiliar system and understanding at a glance what it's doing. It is far more annoying to predictably deploy and tightly couples your application configuration to the host system configuration. (Even moreso if you created dedicated users for each application, which I understand is the recommended solution.)

If I'm just holding it wrong and there exists some better tooling to manage podman in prod that I don't know about, I'm happy to hear about it.

[throwaway894345]

Honestly with the hell I'm going through just trying to get Podman to run properly on macos, I can't imagine trusting the Podman people with a production deployment. I was not particularly impressed with the Docker tooling, but Podman has been even worse. This is a not-remotely exhaustive list of things I've run into in the last 24 hours:

* Podman fails to build a 16GB container image (after 30 minutes of downloading dependencies) despite having 90GB free out of a 200GB podman virtual machine

* Podman machine will, for reasons I don't understand, create a filesystem in a block device with wildly different sizes, and it seems like it's just random

* Pushing podman images to a container image registry via the Podman Desktop UI gives no indication that it's doing anything or even recognized the "push image" click, a success or error notification _might_ appear several or tens of minutes later or possibly not at all

* Starting a podman machine might work, but it fails ~75% of the time with not-particularly-exotic options (a bunch of ram and disk) and very cryptic error messages, frequently telling me to file bug tickets (I have)

* Podman Desktop won't let me create a podman machine with more than 44GB of disk, but the podman machine CLI won't let me create a machine with fewer than 100GB (IIRC--it's some number larger than 44, in any case)

Apart from the container image being absurdly large (Python developers love massive packages, I guess), I'm not doing anything exotic.

[frje1400]

There's a reason why Orbstack can charge money: it just works, and podman doesn't (on macos).

[stryan]

> Having your whole application with its containers, volumes, and networks all defined together in one easy-to-read YAML file is a way better experience. Deployment is two steps: 1. `git clone foo` 2. `docker compose up -d`. You can see the state of the application containers with `docker compose ps`. You can run multiple compose applications on the same host and manage them separately by putting them in different directories.

I always felt it the other way around: docker compose files are weird blobs of YAML that I have to hunt down the location of or parse their under-speced labels to find the location of. I can't make them depend on any non-container services[0], the break my firewall rules[1], and I have to use a whole mess of bespoke tooling just to do normal start/stop/restart operations with them instead of using the same commands I use for literally any other service.

> With quadlets, you delegate everything to systemd. You have to break the configuration up into a bunch of tiny unit files and then separately copy them to /etc or a dedicated user's dotfiles.

The nice thing about quadlets is exactly that, they integrate with systemd and by extension the rest of the system. I don't have to think about `webapp.container` as a "Docker container" I can think of it as just `webapp.service`, like any other piece of software I would install and run. All the related files are in one of the well-speced file locations that follow the same hierarchy as anything else on the system (user -> etc -> /usr), optionally grouped in folders[2].

> Good luck SSH'ing into an unfamiliar system and understanding at a glance what it's doing.

Just use the same tools you'd use on any other systemd system: `systemctl list-units`, `systemctl status`, etc. Versus having to hunt down compose files either manually or by parsing the under-specified labels on the containers.

> (Even moreso if you created dedicated users for each application, which I understand is the recommended solution.)

TBH I've rarely seen this advice. Most people I know just run it as root (which is what I do) or as a `podman` user. But even in this situation it should be pretty easy to figure out whats' running, as you know it's all running as one user and is hard-namespaced to only rely on resources available in that account.

> If I'm just holding it wrong and there exists some better tooling to manage podman in prod that I don't know about, I'm happy to hear about it.

Quadlets are just files that created systemd services, so basically any configuration management or deployment tool will manage them fine. Ansible has a dedicated Quadlet role that works pretty well, or just git clones+`systemctl start`. This would probably be the recommended way if you're not using k8s/etc.

Alternatively, you can just `git clone /etc/containers/systemd/`, `systemctl start container` like with docker compose. If you're running multiple containers, either refer to them with `Wants=`/etc in the Quadlet files, create a `.target` file that references them all, or put them all in a `.pod` and start the pod. I think this is the part were most people stumble though: when you're used to treating containerized software as a separate kind of "thing" it's a little weird to go back to treating it like normal services.

I've been writing something to help with deploying quadlets GitOps-style[3] that will hopefully fill the "more than one server but less than kubernetes" deployment gap.

[0] Unless I wrap the compose steps in a systemd unit, at which point now I have two problems.

[1] Caveat, this has probably gotten better overall but I still run into compose-related firewall issues about once or twice a year

[2] The newer versions of Podman also support `.quadlets` files, that merge all the quadlets into one file.

[3] https://github.com/stryan/materia . There's also https://github.com/orches-team/orches and https://github.com/ubiquitous-factory/quadit

[stevenhuang]

Parent complaint sounds like they are just unfamiliar in general with managing regular system services using unit files/systemctl.

[stryan]

Yeah, I wouldn't be surprised if a lot of the mindset differences come down to people used to using Docker Compose as a development environment being uncomfortable with managing things on a real/traditional/production/whatever-you-want-to-call-it server. Compose treats things as sort of a hermetically-sealed "Application" versus a collection of services. Quadlets are more the latter, and of course that's all Docker Compose is as well but it's a decently good abstraction over it.

[DeathArrow]

> I think perhaps that's less true for running a web service (where the linux environment is just a means to that end).

If some BSD would support OCI containers, I would run my apps on BSD.