[dijit]

Before this topic boils down to: SystemD gave me cancer/SystemD cured my aids as it always seems to. Please permit me get my opinion across (as it is more nuanced) before I continue.

I sincerely believe that systemd solved a problem that nobody was willing to solve, and it has every right to solve those problems any way it wants (when you ask for help you don't get to choose how you are helped after all) - my concerns boil down to the fact that the adoption it has seen has lead indirectly (or, directly) into a monoculture; and a monoculture that almost certainly will stifle innovation since a replacement will need to be bug-for-bug compatible. Prior init's were actually quite easy to replace and alternatives were often used, but these days most software assumes systemd and this situation gets worse every year.

That said; and with the knowledge that while I am afraid of systemd as monoculture (and a relatively opaque one); this line "systemd, as a service manager, is not actually a bad piece of software by itself. The fact it can act as both a service manager and an inetd(8) replacement is really cool."

Is something I vehemently disagree with.

For starters, (x)inetd is an anti-pattern, everything I understand about systems development indicates we should be seperating concerns as much as possible, having one super-server that launches everything under one daemon is directly opposed to this.

"But", I hear you thinking already: "systemd runs services as independent users, it solves that!", and I would agree with you, except now pid 0 is listening to the network instead.. That doesn't strike me as much better.

If there's a bug/backdoor in your binary distributed version of systemd then you are SOL and your whole system is owned as root, but at least a bug in your application might not expose your entire inetd user. :\

It's also a common issue that inetd's architecture can lend itself to getting DoS'd harder than other more-standard daemons, except now it's your pid 0 being DoS'd; not sure how you recover from that honestly.

EDIT: if you are going to downvote, please provide reasons. Sick of this holy war, lets just end it reasonably please.

[awilfox]

The monoculture problem is one that I really tried to cover in the article, indeed.

While I do agree that a "super server" listening to the network could be a new (or old, with (x)inetd) concern, a well-written and well-designed pid1 doing this isn't much more risky than having the services manage themselves. The listener could/should be just as unprivileged as the target daemon, nominally using the same uid/groups and root directory as the target as well. And indeed, systemd's .socket files support the same environment control variables (User=, Group=, SupplementaryGroups=, RootDirectory=, WorkingDirectory=, et al) that services do.

So that boils down to "are people writing socket activation units correctly", which is probably "no", but could be "yes".

[znpy]

> The monoculture problem is one that I really tried to cover in the article, indeed.

The issue is that the previous "multi-culture" really sucked real bad. It was a shit show, really.

systemd is rising the bar by A LOT in terms of systems management, and most alternatives are simply not keeping up.

But can you really blame it on systemd making a stellar job on its own?

If anything, we could blame it on the "alternative projects" doing a fairly poor job and delivering very little.

edit: systemd is so good that FreeBSD people have already started pondering if they should build something similar for themselves: https://www.youtube.com/watch?v=o_AIw9bGogo

[dijit]

Fuuuuuck I hate that talk by Benno.

Mostly it strawmans the opposition, says its because they dont like change primarily.

Then he indicates that macos’s launchd is an improvement, despite it being one of the most painful parts of advanced macos administration.

Its a talk that comes up often in these threads but it completely betrays the idea of having a fruitful dialog about the pros and cons of this design and the situations where it can be beneficial.

[acdha]

> Then he indicates that macos’s launchd is an improvement, despite it being one of the most painful parts of advanced macos administration.

This seems like a rather hot take. Most of the Mac admins I knew when I worked in the field were quite happy to have a single standard way to solve that class of problems.

[ptx]

Why is inetd an anti-pattern? Isn't separating all the work of opening ports and launching daemons into a separate component (so that it's not mixed into every other component) exactly what separation of concerns is all about?

[pi-e-sigma]

Inetd approach is just fine. Don't listen to to the worshippers of systemd complexity. It's like listening again the guys who insisted that JEE is the future and you are stupid because you prefer simple solutions to simple problems.

[Nullabillity]

> Prior init's were actually quite easy to replace and alternatives were often used, but these days most software assumes systemd and this situation gets worse every year.

Not really. Sysvinit scripts are... scripts full of sysvinitisms (double forking and PID files, anyone?). Idiomatic systemd daemons are, comparatively speaking, very straightforward. Of course, there are a lot of nice-to-have features, but to get a running system, you should largely be able to get away with parsing Wants=, After=, and ExecStart=.

[einpoklum]

> Not really. Sysvinit scripts are... scripts full of sysvinitisms (double forking and PID files, anyone?).

... well, a bunch of scripts are something that's relatively easy to replace. It's not as though other system components have reliance of these scripts and their sysvinit'isms baked in.

From what I've heard (though not verified) - non-systemd distributions seem to manage to work with upstart, or openrc, instead of sysvinit, without much hassle.

[dijit]

Previous inits were glorified loop { fork() } programs.

There were many scripts, many implementations, but interestingly most were compatible with each other.

The largest issue inits had was that the flexibility they provided gave distro maintainers a lot of choice in how those scripts should operate to be consistent with the rest of the system.

Things like log locations.

The issue to be solved is: how to determine parallelism in boot, how to supervise a process with minimum complexity, and how do we do structured logging.

Interestingly: SMF solved all of these a decade before.

[einpoklum]

> Previous inits were glorified loop { fork() } programs.

Maybe (I'm not an expert on old init systems), but current inits aren't that.

> The issue to be solved is: how to determine parallelism in boot

I think you're replying to another comment of mine. At any rate, there were and are alternative solutions rising to meet this challenge - which involve nothing like the behemoth which is systemd. Examples: OpenRC, runit, GNU Shepherd (sort of).

https://en.wikipedia.org/wiki/OpenRC

[einpoklum]

> I sincerely believe that systemd solved a problem that nobody was willing to solve

What problem was there which nobody was willing to solve?

Also, "solving" a problem by creating a mechanism that in itself highly problematic in other ways does not necessarily count as a solution; it is a shifting-around of problems.

-----

The systemd "holy war", such as it is (mostly complaints on forums and in blog posts and in personal chats; don't remember any violence in this war) - is due to four reasons:

* The significant problems which systemd introduces.

* The way systemd has been developed and managed as project (including some grievances with individuals).

* The fact that distributions have not only adopted systemd, but made it effectively impossible to opt out of.

* Faults with the process in which systemd was adopted as a required default by all of the main distributions - bypassing wide opposition without addressing its criticism. So, the adoption of systemd exposed technical-governance/power-dynamics problems in the Linux distro world, with systemd serving as the symbol for those.

[awilfox]

I like the off-by-one there, and choose to interpret it as intentional.

The fourth item is what I find to be the most interesting. I really hadn't thought about it that way before. I remember the discussion in Fedora mailing lists around systemd and iirc the criticisms were mostly actually answered and handled. But in Debian it really wasn't and it fractured the community a lot.

It is perhaps the case that the "anti-systemd" crowd wouldn't be "anti-systemd" if they felt like they had been heard. Perhaps that's the bigger lesson we should all be learning: to listen closely and respond respectfully.

[einpoklum]

> I like the off-by-one there, and choose to interpret it as intentional.

Amongst our weaponry are such elements as... I'll come in again.

https://www.goodreads.com/quotes/1495-nobody-expects-the-spa...

> It is perhaps the case that the "anti-systemd" crowd wouldn't be "anti-systemd" if they felt like they had been heard.

I would say it is the other way around. systemd would not have been adopted as the default init system, had the anti-systemd crowed not been ignored.

But that's not even the main point. The grievance is not about how some individuals did not "hear". It is about the _possibility_ of such a decision being taken with that level of technical-community resistance; i.e. the expectation is that decent process would not depend only on the benevolence of those in charge. That's why it's a structural rather than a personal failure IMNSHO.

And again, there's the conflation, or bundling, of the multiple decisions:

1. Offer systemd in the distribution

2. Make systemd the default option for the distribution

3. Necessitate installation & use of systemd with the distribution

I am specifically pretty certain that Devuan would never have been forked if systemd were merely a configurable installation option in Debian.

[throwaway7356]

> I am specifically pretty certain that Devuan would never have been forked if systemd were merely a configurable installation option in Debian.

You can't stop 5 or so random people on the internet from starting the 100th irrelevant Debian-based derivative.

[tuna74]

"* The fact that distributions have not only adopted systemd, but made it effectively impossible to opt out of."

The job of a distro is to decide what to ship and integrate that in the best way possible. Would a distro let you choose between glibc and musl for example? If you want either, use a distro that chooses that.

[awilfox]

This has more to do with systemd trying to be a "system layer" project for Linux, and influential projects (initially Gnome, then KDE, and now a lot of other things) going along with it.

Honestly, I see no reason a distribution couldn't support multiple service managers. Indeed, Gentoo officially support OpenRC and systemd and do just fine. However, you need to have the resources to do it and the desire. I think there are multiple distributions with one or the other, but Gentoo is likely the only with both.

[einpoklum]

> However, you need to have the resources to do it and the desire.

If resources were lacking to offer both regular and systemd-based init and service management, than a systemd offering should have been deferred until such time when resources become available, or different projects did some work of their own to reduce the amount of necessary resources. Offering multiple other init/service management options apparently doesn't require many resources.

(Of course, they could have saved a lot of resources by dropping GNOME until the GNOME people started being civil and not hard-depend on systemd... but I realize that's a bit of a controversial suggestion :-P )

[throwaway7356]

> systemd offering should have been deferred until such time when resources become available

Why would you delay adoption of a technical superior solution if there is no interested/lack of resources in continuing maintenance of the inferior solution?

Distributions did not stop adopting Python 3 and invested in continuing Python 2 either.

[einpoklum]

> Why would you delay adoption of a technical superior solution if there is no interested/lack of resources in continuing maintenance of the inferior solution?

But a technically superior solution was not on the table to become the default, the suggestion was to use systemd and effectively prevent the use of anything else (including potentially technically superior solutions).

[acdha]

> For starters, (x)inetd is an anti-pattern, everything I understand about systems development indicates we should be seperating concerns as much as possible, having one super-server that launches everything under one daemon is directly opposed to this.

It means that this server had to be carefully secured but the benefit is that you have exactly one bit of heavily-audited code listening to the network, logging activity & problems, starting processes, changing users / dropping privileges, setting up namespaces, etc. I’ve seen a lot of code get various combinations of those wrong so I think that’s a far more nuanced problem than in your portrayal.

Those are all things people should know how to do but I’ve seen Java or PHP running as root in production because someone couldn’t figure out how to drop privileges needed only at startup enough times to appreciate the benefits from systemd making it so much easier to do things right.

[sevagh]

>"systemd runs services as independent users, it solves that!"

It's in a terrible way, too. Not sure it improved but I recall somebody once, at a company I worked at, trying to introduce per-unix-user systemd services (e.g. `graphite` user for running Graphite), and there were some atrocious steps required like `loginctl --enable-linger` and God knows what.

We moved to running everything systemd as root, it's easier (and you can specify which unixuser the actual systemd unit runs at, which is "close enough").

[Nullabillity]

Loginctl/per-user systemds are for managing interactive sessions. System services with User= is exactly the way you'd go for isolating background daemons.

[ptx]

What if you want to let a non-root user manage the systemd unit for a particular service? Is there a better way than per-user systemd instances?

[acdha]

Check example 3.4 here for a polkit policy allowing an arbitrary user to restart a single unit:

https://wiki.archlinux.org/title/Polkit

[twic]

Does this let them modify the unit file, create units, create timers, etc?

User systemd allows that kind of complete self-service, and so lets you do application deployment and management without touching the root account, which is rather nice.

[BenjiWiebe]

If they can edit a service that runs with more privileges than they have, that's a privilege escalation vulnerability.

Polkit lets a non-root user restart a root/privileged service without letting the non-root user gain privileges.

[chuckadams]

chgrp, setfacl, sudo.conf ... lots of choices.

[rst]

"Nobody was willing to solve" is a bit of a stretch when upstart got shipped at pretty much the same time. (Sponsored by the commercial home of a particular Linux distribution -- but so was systemd.)

[acdha]

I used Upstart a lot but it did less and had various unfixed bugs whose failure mode was becoming unmanageable. I give Ubuntu credit for starting it but was reminded of jwz’s CADT rant periodically until we upgraded to a release which used systemd instead.

[Karellen]

> my concerns boil down to the fact that the adoption it has seen has lead indirectly (or, directly) into a monoculture;

Were there multiple implementations of sysv-init being used by different distros on Linux before systemd came along?

[dijit]

yes, at least 5.

sysvinit, s6, openrc, runit and Solaris SMF

[Karellen]

Were they alternate implementations of sysv-init? Or did they do their own thing, and also happen to run sysv-init scripts for back-compat? Because systemd also runs sysv-init scripts for back-compat.

Edit: Also, were any distros actually shipping those as supported init systems? I was under the impression that most of them were still in the "experimental" stage and not viable replacements (yet).

[liveoneggs]

OpenRC and Solaris SMF definitely did their own thing. SystemV-init and the "run levels" concept was flawed from the very start when it replaced the previous BSD-style "single user" vs "multi user" boot up.

Linux only adopted SystemV style init because it was the norm in Solaris at the time. It is not a linux-ism.

As this article notes - systemd is actually fine as an init system and hardly anyone denies it. It's all of the other stuff (journald, resolved, timers, etc) + tight coupling + environmental assumptions that is the problem.

[awilfox]

I'm unaware of any Linux distribution using SMF. If there was, I would have been running it. I didn't really like pfexec (it felt bolted on and not quite there yet), and I really really hated ZFS (and still do), but SMF made Solaris administration a joy.

I don't think any distribution was using runit before systemd, but it was available in Gentoo as a sysvinit replacement and ISTR it was used in the Rails community for supervising Unicorn.

[teddyh]

> now pid 0 is listening to the network

Pid 1, surely.