[razighter777]

What practical problems do you run into with systemd?

All the compliants I see tend to be philisophical criticism of systemd being "not unixy" or "monolithic".

But there's a reason it's being adopted: it does it's job well. It's a pleasure being able to manage timers, socket activations, sandboxing, and resource slices, all of which suck to configure on script based init systems.

People complain in website comment sections how "bloated" systemd is, while typing into reddit webpage that loads megabytes of JS crap.

Meanwhile a default systemd build with libraries is about 1.8MB. That's peanuts.

Systemd is leaps and bounds in front of other init systems, with robust tooling and documentation, and despite misconceptions it actually quite modular, with almost all features gated with options. It gives a consistent interface for linux across distributions, and provides a familar predictible tools for administators.

[rendaw]

I wrote up some issues with service reliability here https://github.com/andrewbaxter/puteron/?tab=readme-ov-file#...

Design-wise, I think having users modify service on/off state *and* systemd itself modify those states is a terrible design, which leads to stuff turning back on when you turn it off, or things turning off despite you wanting them on, etc. (also mentioned higher up)

FWIW after making puteron I found dinit https://github.com/davmac314/dinit which has a very similar design, so presumably they hit similar issues.

[BadBadJellyBean]

Systemd usually only modifies the state if is somehow configured to do so. Socket activations, timers, depwndencies. They all tell systemd what to do and can usually be modified if needed.

[palata]

> But there's a reason it's being adopted: it does it's job well

My problem with systemd is that it's taking over more and more and locking in. It is encouraging developers to have a hard dependency on it, and making it harder to have an alternative.

My problem is not philosophical with "it's a monolith, it's not unixy". My problem is "it's on the way to lock me in".

We like to complain about lock-in across the board. I don't see why it would be different with systemd.

[BadBadJellyBean]

I think you got it backwards. Systemd is a standardization that is appealing to developers. They want to adopt it because it makes their life easier. It is just nice to know that all the tools you need for a system are there and work together. Pluggability is hard to maintain and is only done if there is no standardization.

I somehow don't think your gripe is with systemd but with developers who prefer the easy route. To be honest though you get something for free. If you want it differently then you have to do it yourself.

[palata]

I don't think it's backwards; it's not incompatible with what you said.

> It is just nice to know that all the tools you need for a system are there and work together.

It is indeed! Just like everybody uses WhatsApp for a reason. But because everybody uses WhatsApp, it is very difficult to get traction with an alternative. That's the lock-in part.

It is easier for developers to only care about systemd. It's often worse: many times I have seen projects that only work with Ubuntu. Of course I understand how it was easier for the developers of those projects to not learn how to "be nice" and "do it right". That does not mean I should be happy about it.

> If you want it differently then you have to do it yourself.

Or I should support alternatives, which I do. I am not saying you are not allowed to use systemd, I am just explaining why I support alternatives. Even though systemd works.

[BadBadJellyBean]

I'd argue that "do it right" and "be nice" are incredibly subjective. I'd say they were already nice enough to write open source software. And I don't think it is wrong to to write what you want to write.

The comparison with WhatsApp has a a huge flaw: WhatsApp is not LGPL licensed software. No one can really take systemd away. There is very little risk in depending on it apart from less choice. But I already argued that the expectation of choice in free software is a big ask.

And there is no one stopping anyone from implementing systemd's api surface.

The reason why I say you got it backwards is that you are against systemd making available all their tools when in reality it is the distro maintainers choice to use them and the developers choice to depend on them. Most of systemd is optional and nothing prevents developers from writing abstractions. But the simple truth is that systemd is offering a compelling value that people are just accepting.

[its_magic]

> Systemd is a standardization that is appealing to developers. They want to adopt it because it makes their life easier. It is just nice to know that all the tools you need for a system are there and work together. Pluggability is hard to maintain and is only done if there is no standardization.

That's the official story, but like most official stories, it doesn't really hold up to scrutiny.

I built an entire system from scratch with over 1,500 packages installed. Everything under the sun. Works just fine with sysvinit. Completely seamless.

If KDE/Gnome can't figure out how to fit in with the overall system design the same way thousands of other packages somehow manage to do, then their services are no longer required. Good riddance to their bloated asses. I prefer to invest my CPU cycles in better software.

Init scripts for services and such are properly handled by the distro maintainer (packager), not the developer, although it's always nice to see examples he's provided to help guide the development of my preferred version.

[BadBadJellyBean]

I am honestly happy for you that you made your system the way you want it. That is a good thing and please keep doing what you are doing.

This is not relevant to the average user. The average PC user doesn't use Linux and the average Linux user uses an off the shelve distro. For these distros it is very attractive to have a bunch of core services ready that work together because they are released as one. It can be done but why the hassle? What is the upside for the maintainer apart from maybe the moral high ground?

Software projects can also benefit from standardization. They can concentrate on writing functionality instead of maintaining abstraction layers. And I believe the more mainstream distros choose the SystemD stack the more it becomes the default or at least the initial implementation for their software.

We also have to keep in mind that this kind of standardization is nothing new. Pretty much every distro depends on the GNU coreutils. Maybe not on the binaries themselves but at least on their API. That is not very different from SystemD. We have a POSIX standard.

Final word regarding sysvinit: I worked with sysvinit, upstart and systemd and having an opinionated config format for services is so much better, in my opinion. Not having to read a whole shell script to know how a service works is such an improvement and the easy overrides for units (for example distro packaged ones) is amazing.

Note: In my post I counted distro maintainers as developers.

[its_magic]

You lost me when you started talking about the average user. I don't care about that guy or his desires. At all.

I miss the days when computing was about the above average guy--not the simpleton who needs his hand held, so everything has to be dumbed down to the lowest level to suit him.

Heard it all before, and I'm not interested in anything systemd has to "offer." Especially all the bugs and security issues.

This distro isn't for you. That's OK. systemd, and wayland, etc that some are so excited about isn't for me or a number of others, and it will never be. We are going our separate way. Just look at all the comments below. Lots of upvotes too.

[theamk]

It's not a lock-in as much as making a much better product.

For example, I never liked the idea of having my programs to manually daemonize, manage logs, set up permissions and all that boring, but security-critical stuff. And with systemd, I don't have to! Program reads from stdin/stdout, maybe gets socket from socket activation, and systemd does the rest.

Is it lock-in? Only because other system suck. Like, seriously, what stopped xinetd from having rudimentary volatile "on/off" control, so I could disable misbehaving service? Or why is start-stop-daemon so _stupid_, discarding all startup error messages? And don't get me started on all the different init file dialects for each system.

Maybe if the sysvinit programmers actually cared about providing nicer services to app developers, we would never end up with systemd.

[its_magic]

The problem with the word "sysvinit" here is it's sort of a red herring. BSD init is better, in my opinion. I don't like managing all those symlinks. Plus, sysvinit is an old 90s application and its code does have some cruft built up over the years that could be removed and simplified. I'm devising a new init for my system that's much simpler than sysvinit and much closer to BSD.

[theamk]

"BSD init", "much simpler"... So does this mean you still expect applications to manage their own logs, daemonization and security setup themselves?

If yes, that's yet another init system not made for application writers.

[its_magic]

Manage their own logs, daemonization, and security? The humanity! How will they ever manage all of that?

Come on man. It's been done for decades.

It doesn't take a giant bloated infrastructure to manage most people's needs, which are quite basic in most cases.

[theamk]

.. and that opinion is a great explanation of why systemd won.

Turns out, a lot of people are not happy with "Come on man. It's been done for decades." attitude, and they wanted something new and much better. And so when something new came up, they jumped on it with both feet.

It's instructive to read Debian CTTE discussion on init systems (btw I think it's best tech drama of 2013, highly recommend) - a lot of people dismissed the sysvinit early on because it had no features (example [0]), which means the choices were either upstart and systemd. And between two of those, systemd is a clear win.

Read the thread and look at how many highly technical people with no relation to Fedora or Poettering is ready to choose _anything else_ just to get away from "it's been done for decades".

[0] https://lists.debian.org/debian-ctte/2013/12/msg00234.html

[cyberax]

Ohh... I have sooooo many issues with systemd. The core systemd is fine, and the ideas behind it are sound.

But it lacks any consistency. It's not a cohesive project with a vision, it's a collection of tools without any overarching idea. This is reflected in its documentation, it's an OK reference manual, but go on and try to build a full picture of system startup.

To give you concrete examples:

1. Systemd has mount units, that you would expect to behave like regular units but for mounts. Except that they don't. You can specify the service retry/restart policy for regular units, including start/stop timeouts, but not for mounts.

2. Except that you can, but only if you use the /etc/fstab compat.

3. Except that you can not, if systemd thinks that your mounts are "local". How does it determine if mounts are local? By checking its mount device.

4. Systemd has separate behaviors for network and local filesystems.

5. One fun example of above, there's a unit that fires up after each system update. It inserts itself _before_ the network startup. Except that in my case, the /dev/sda is actually an iSCSI device and so it's remote. So systemd deadlocks, but only after a system update. FUN!!!

6. How does systemd recognize network filesystems? Why, it has a pre-configured list of them: https://github.com/systemd/systemd/blob/4c6afaab193fcdcb1f5a... Yes, you read it correctly. A low-level mount code has special case for sshfs, that it detects by string-matching.

7. But you can override it, right? Nope. This list is complete and authoritative. Nobody would ever need fuse.s3fs . And if you do, see figure 1.

I can go on for a looooong time.

[nomel]

5 and 6 sounds like good candidates for a bug reports/PR, if there's not already some "right" way to do it.

[cyberax]

They're already reported. And ignored. Have you _seen_ the systemd issue backlog?

The iSCSI loop issue: https://github.com/systemd/systemd/issues/34164 It keeps popping up again and again and is summarily ignored.

The remote FS detection also came up multiple times, and the maintainers don't care.

[nomel]

> and the maintainers don't care.

I'm not sure that's fair. I think better proof of this would be a rejected PR rather than a neglected bug report.

This is Linux, after all. Problems found with specific hardware are almost always solved by people with that hardware, not the maintainers, who are usually busy with the 99%.

[cyberax]

The problem here is more fundamental.

Lennart refused to make all the /etc/fstab options available in regular mount units. And yes, there was an issue, no I'm too tired to look for it. The wording was pretty much: "Give up, and gtfo, this is not going to happen. Just because."

I'm convinced that systemd can't be fixed by its current team of maintainers. They are just... untidy.

I don't know about you, but if I end up writing low-level code that _needs_ to know whether the mounted file system is "remote", I won't do that by comparing against a hard-coded list of filesystems inside PID0. Or by using wild heuristics ("if it's on a block device, then it's local").

I would put these heuristics in a helper tool that populates the default values for mount units. Then allow users to override them as needed. With a separate inspector tool to flag possible loops.

[dTal]

This is one example of a more general complaint about systemd and related projects: they force policy, rather than simply providing mechanisms.

I recently did a deep dive on my laptop because I was curious about an oddity - the /sys file to change my screen backlight (aside, why /sys and not /dev anyway?) was writable only by root - yet any desktop shell running as my user had no problem reacting to brightness hotkeys. I wondered, how did this privilege escalation work? Where was the policy, and what property of my user account granted it the right to do this?

It turns out the answer is that the desktop shells are firing off a dbus request to org.freedesktop.login1, which is caught by systemd-logind - or elogind in my case, since I do not care for systemd. A login manager seemed an odd place for screen brightness privilege escalation, but hey if it works whatever - it seemed like logind functioned as a sort of miscellaneous grab bag of vaguely console-related stuff. Generally speaking, it consults polkit rules to determine whether a user is allowed to do a thing.

Not screen brightness, though. No polkit rules. Nothing in pkaction. logind was unilaterally consenting to change the brightness on my behalf. And on what grounds? It wasn't documented anywhere so I had to check the source code, where I found a slew of hardcoded criteria that mostly revolve around physical presence at the machine. Want to change screen brightness over ssh? Oh but why would you ever want to do that? Hope you have root access, you weirdo.

I removed elogind. A few odds and ends broke. But nobody tells me what to do with my machine.

[simoncion]

> I think better proof of this would be a rejected PR rather than a neglected bug report.

I understand the sentiment you're expressing here, and it's often a reasonable one.

However, when every sharp edge case I've encountered with SystemD (both professionally and personally) ends either in a open Github Issue whose discussion from the project maintainers ends up being "Wow. That's tricky. I'm not sure whether or not that behavior is correct. Maybe we should do something about this or document this so other folks know about it." (and then nothing happens, not even the documentation) or a closed Github Issue with "Sorry, your usecase is <strike>inconvenient to implement</strike> unsupported. E_NOTABUG", expecting PRs is expecting way too much.

[its_magic]

I've long been in the habit of reading accounts like yours, understanding the truth and wisdom that's being expressed, then noping the fuck out of the tech/product/situation in question. It has saved me a lot of trouble over the years. Even as others are completely mystified. Some people just like abuse, I guess.

"Sweet dreams are made of this..."

[jdub]

OK, think it through...

How do we determine that a specific instance of a filesystem mount is "remote", or even requires a "network"? Consider that the network endpoint might be localhost, a netlink/unix/other socket, or, say, an IP address of the virtual host (practically guaranteed to be there and not truly "remote").

systemd has .mount units which are way more configurable than /etc/fstab lines, so they'd let you, as the administrator, describe the network dependency for that specific instance.

But what if all we have is the filesystem type (e.g. if someone used mount or /etc/fstab)?

Linux doesn't tell us that the filesystem type is a network filesystem. Linux doesn't tell us that the specific mount request for that filesystem type will depend on the "network". Linux doesn't tell us that the specific mount request for that filesystem type will require true network connectivity beyond the machine itself.

So, before/without investing in a long-winded and potentially controversial improvement to Linux, we're stuck with heuristics. And systemd's chosen heuristic is pretty reasonable - match against a list of filesystem types that probably require network connectivity.

If you think that's stupid, how would you solve it?

[cyberax]

> How do we determine that a specific instance of a filesystem mount is "remote", or even requires a "network"?

Like systemd authors do! Hard-code the list of them in the kernel, including support for fuse and sshfs. Everything else is pure blasphemy and should be avoided.

Me? I'd have an explicit setting in the mount unit file, with defaults inferred from the device type. I would also make sure to not just randomly add landmines, like systemd-update-done.service. It has an unusual dependency requirements, it runs before the network filesystems but after the local filesystems.

I bet you didn't know about it? It's a service that runs _once_ after a system update. So the effect is that your system _sometimes_ fails to boot.

> systemd has .mount units which are way more configurable than /etc/fstab lines

It's literally the inverse. As in, /etc/fstab has _more_ options than native mount units. No, I'm not joking.

Look at this man page: https://www.freedesktop.org/software/systemd/man/latest/syst... The options with "x-systemd." prefix are available for fstab.

Look for the string: "Note that this option can only be used in /etc/fstab, and will be ignored when part of the Options= setting in a unit file."

[jdub]

Sounds like your admin, distro, or the systemd team could pay some attention to systemd-update-done.service

The "can only be used in /etc/fstab" systemd settings are essentially workarounds to do those things via fstab (and workaround fstab related issues) rather than depend on other systemd facilities (c.f. systemd-gpt-auto-generator). From a "what can you do in /etc/fstab without knowing systemd is working behind the scenes" point of view, then yes, systemd units are vastly more configurable.

[simoncion]

> How do we determine that a specific instance of a filesystem mount is "remote", or even requires a "network"?

The '_netdev' option works a treat on sane systems. From mount(8):

       _netdev
           The filesystem resides on a device that requires network access
           (used to prevent the system from attempting to mount these
           filesystems until the network has been enabled on the system).

It should work on SystemD and is documented to in systemd.mount

  Mount units referring to local and network file systems are distinguished by their file system type specification. In some cases this is not sufficient (for example network block device based mounts, such as iSCSI), in which case _netdev may be added to the mount option string of the unit, which forces systemd to consider the mount unit a network mount.

but -surprise surprise- it doesn't reliably work as documented because SystemD is full of accidental complexity.

[jdub]

Sure, and systemd would translate that directly into a dependency on network startup, which is precisely equivalent to the approach I mentioned that depends on operator knowledge. It's configuration, not "just works" inference.

[jcarrano]

That is one of my problems with systemd: it has way to much "magic" built in. SysVinit/OpenRC and related are easy to understand and debug: they only do what's in the scripts.

[SuperNinKenDo]

I love systemd, but you've hit on one of my biggest complaints. The mounting promises a cohesive system and instead gives you a completely broken mess, with mounts being split across .mount unit files, fstab, and worst of all, .service unit files. It's a totally incoherent mess, and that's only _after_ you figure out why nothing is working right, and build a complex mental model of every single feature that does or doesn't work in which scenario. Knowledge you only gain after screaming and tearing your hair out for a weekend. Your reward? A totally incoherent, inconsistend mess.

I hate mounts in systemd.

[cyberax]

And don't forget automounts! They are so much fun!