[qplex]

Has anyone had positive experiences with systemd? Success stories because of feature x?

I've pretty grim personal opinion of it, based on similiar experiences as this story. I'd rather just use a simpler approach (another init) that _works_, and doesnt involve hours of debugging. I've never had any troubles with sysv-style init for example.

[Mister_Snuggles]

Yes.

I used to run my home automation stuff (Home Assistant, Node-RED, and a custom shim between my power meter interface and MQTT) in a FreeBSD Jail and recently switched to a Debian VM.

Getting everything to start properly with FreeBSD's rc scripts was a minor nightmare. It took a lot of work to accomplish the task of "Run command x, which doesn't daemonize, as user y in directory z and redirect stdout and stderr to blah.log". For quite a while, I was just running things by hand under tmux because I couldn't figure out the exact method to make it work properly.

In comparison, doing the same in systemd was incredibly easy. This type of service is really easy to set up in a systemd unit file.

I'll admit that a lot of the problems with doing this in FreeBSD were likely my own lack of skill and/or understanding. Systemd doesn't require this level of skill/understanding to set up a simple service though.

In terms of benefits of systemd, the best one for me is logging. Having stdout/stderr from these processes go to the journal automatically is quite nice and querying it with journalctl is easy now that I've figured out how to use it.

When developing simple services (like the custom shim mentioned earlier), systemd greatly simplifies things since I don't need to worry about daemonizing, switching users, logging, etc.

[fanf2]

Isn't that one command on FreeBSD?

  chdir z && daemon -u y -o blah.log x

https://www.freebsd.org/cgi/man.cgi?query=daemon

[Mister_Snuggles]

I seem to remember having problems with daemon, but I can't remember what they were exactly. The Jail is gone now, so I can't look at the end result to see what I ended up with. By all accounts, this command should do exactly what I need though.

I probably ended up with something like this[0], which doesn't use daemon. Home Assistant also provides a FreeNAS example[1] which uses daemon.

For comparison, the systemd unit file that I ended up with is:

    [Unit]
    Description=Home Assistant
    After=network.target
    
    [Service]
    Type=simple
    Restart=always
    ExecStart=/home/hass/homeassistant/bin/python /home/hass/homeassistant/bin/hass
    WorkingDirectory=/home/hass
    User=hass
    Group=hass
    
    [Install]
    WantedBy=multi-user.target

[0] https://gist.github.com/damoun/96add58f60572cb12c11

[1] https://www.home-assistant.io/docs/installation/freenas/

[GordonS]

At first I was skeptical, but really that was because of systemd being such a big change, when I've already worked with sysv for so long.

Now though, I really like the systemd service approach - it's so much simpler than hacking custom bash scripts, and it's actually really fully-featured too. For example, it supports lots of different service lifecycles, and even has a watchdog included that can restart failed processes.

I'm still less sure of some of the other parts, like journald, but I'm a systemd convert for services at least!

[pfranz]

Over the years I've hit issues where the daemon using traditional init will inherit the environment it gets launched from. So on boot it works as expected, but if a user restarts the daemon it can get subtle behaviors because of that user's environment. For this reason I've been itching to adopt an event-based init for many years (Upstart, being event-based, also addressed this).

I've had frustrations with systemd, but the few times I've wanted to write an init script I was glad it was trivial instead of cobbling something together for traditional init.

[fanf2]

The `service` command fixes problems with environment pollution, and the `start-stop-daemon` command makes init scripts easier to write.

[pfranz]

Of course. The fixes weren't unique to systemd. Upstart addressed those issues too. The problem, either branding or adoption, is that after literally decades it hadn't been widely adopted on sysv-style systems. `/etc/rc.d/init.d/functions` was never ubiquitous enough that you could trust it existed. I've downloaded software that required it on platforms that didn't offer it and meant troubleshooting and rewriting when I should have been on to the next thing.

Maybe because they /could/ be run directly if/when service existed without warning or penalty, it also wasn't widely used.

In practice, I hit it many many times in sysv-style systems. I've used systemd systems for much less time but haven't hit a similar issue since transitioning.

[viraptor]

I'm pretty happy with it as it works (even though I'm not a fan at all of the way it's developed/organised). I was just as happy with upstart though.

No more crazy init scripts, no more pidfiles, better templating, cgroups integration OOTB, better handling of restarts, isolation integration (private tmp, capabilities, ...), etc. I enjoy what systemd-the-init-process provides. I see it as a standardisation of what everyone did in a slightly different custom script into a known key-value format.

[amarraja]

I never had strong feelings about systemd, and the first time I used it I did prefer the service files to sysvinit scripts.

The real wow moment came when discovering socket activation which allowed systemd to take care of zero downtime deployments of web applications - for me that was reason enough to support the project.

Nowadays I'll use a container orchestrator to achieve the same thing, bit it was a good couple of years whilst it lasted.

[IngoBlechschmid]

I didn't follow the systemd vs. sysv war closely, and for a long time was content that I barely managed to remember the new syntax for starting and stopping services when the distributions I used switched to systemd.

In the last couple of days, I became a fan. Not for the many reasons which anyone can look up. But for systemd-nspawn, an awesome tool for spawning lightweight containers.

For instance, you can boot an ephemeral (tmpfs) copy of your system with systemd-nspawn -D / -xb. Several more examples are listed here: https://www.freedesktop.org/software/systemd/man/systemd-nsp...

[qplex]

>for a long time was content that I barely managed to remember the new syntax for starting and stopping services when the distributions I used switched to systemd.

I've felt the same way, but then again I'm not really missing SysV runlevels either.

Generally I don't find myself tinkering with init systems too often, and a simple list of commands to run on startup in-order suffices for most of my needs.

[throw0101a]

> I've pretty grim personal opinion of it, based on similiar experiences as this story.

Many issues with systemd seem not to be with systemd-as-init-replacement, but rather with with systemd-as-kitchen-sink. It's the tight coupling that annoys many people.

Does udevd really need to be in the same repo? While there may be some nice things about journald, does it really have to be in the same source package? (And why can't it support remote logging with the industry standard syslog protocol? Now I have to run journald and rsyslog. And why doesn't it have an ACID file format that doesn't self-corrupt at times? Why couldn't they just use SQLite or OpenLDAP's LMDB?)

[regularfry]

The udev move was what convinced me the whole effort was bad faith. I have yet to be disabused of this notion.

[nailer]

Sure! Look at any StackOverflow post where 'how do I run X as a service on Linux' is now 'make a .service file' rather than a combination of bash, pm2, forever, etc.

Compare the upvotes on the .service answers to the alternatives.

Also earlier today I wanted to find out about systemd timers and the first post was this, which thinks they're way better to debug than cron: https://mjanja.ch/2015/06/replacing-cron-jobs-with-systemd-t... - mainly because they have an options to 'run this job right now, in exactly the same environment' - that's an excellent feature, and something cron should have had ages ago.

Getting an impression of systemd from HN is like getting an impression of politics from Twitter. You'll be exposed to a lot of views, but not a wide variety of them.

[pepemon]

Have you worked with CoreOS? It's fully driven by systemd, no problems at all. You know, when something works, nobody talks about it. But when it fails for just one time...

[kd913]

Im having a lot of success with various aspects systemd. I have happily replaced grub with systemd-boot. It makes my life easier with switching OSes, configuration is simpler, boot times are much quicker and it doesn't load practically another OS at boot time. That and it enables much better boot time information and access to UEFI firmware.

Then there is systemd-networking. So much easier, simpler to configure than netplan.io and network manager. Again, lots of information presented through journalctl and I never have to deal with networking issues.

I have systemd-analyze which gives me a clear picture of boot times and security issues. From my perspective, these kinds of things are a step above what existed before.

[StreamBright]

I am not sure why people do not understand that you could have a single file service descriptor and not merging all system services into one at the same time.

Don't like init style shell scripts? Replace it with something like systemd's service descriptors. It would be fine with me. Now, using this as merge god knows how many different services into one badly written piece of crap that uses 100-1000x more energy than the services it replaces is not an argument that you can win.

[musicmatze]

I use systemd mainly as a user, only on one of my instances I do some more "admin" like stuff. It works and is absolutely reliable for what I do and did by now. I couldn't imagine going to something else and would also see no reason why I should.

[qplex]

I think standardizing init to systemd across distributions has some benefits for everyone.

SysV certainly wasn't and isn't the smooth experience either and took some time to grasp.

[AlexMax]

> Has anyone had positive experiences with systemd?

I was a relatively early adopter to CentOS 7, and I have yet to run into an issue that I could pin on systemd across any of the Linux systems I use or maintain. Then again, my requirements tend to be pretty "normal" and not terribly taxing - at most I crank out a couple of server-specific service and timer files and get on with my life.

I totally understand where some of the critics of systemd are coming from. No matter how you slice it, more lines of code is invariably going to result in more bugs. And the development team does seem a bit myopic at times - though nothing deserving the death threats they get.

However, there were some clear disadvantages to the old-school sysv init, and I thought that it was important that Linux settle on a new standard init system and do it without yet another instance of fragmentation happening, like with rpm vs deb or Flatpak vs Snap or countless other examples. Systemd happened to be it, but honestly I didn't even care what the init system was as long as something finally "won" and didn't introduce yet another schism. In fact, I highly suspect that if something else had won people would have doubtless found an axe to grind with that one too.

[nailer]

> No matter how you slice it, more lines of code is invariably going to result in more bugs.

Is systemd + its services and timer files, bigger than sysvinit, cron, atd, pm2/forever, previous device managers, all the various bash scripts, and everything else it replaces?

I don't know either way. I'm just not completely sure Systemd is bigger.

[rjeli]

But when the bash scripts and pipes break, it’s easier to isolate the broken component vs grepping through the systemd monorepo.

[nailer]

Sure. But bash scripts and pipes break way, way more often then a .service file does.

[geofft]

It's true that a traditional-style init has few bugs on its own, but that means you're moving all the service management, logging, etc. complexity somewhere else, and that gets both buggy and messy. I have many stories of systemd making the overall architecture simpler and thereby both solving bugs and doing things that we had resigned ourselves to not being worth the complexity.

(It's possible other "modern" inits could also do this well, and I do continue to believe that systemd in particular is not the best code even if I agree with its architecture, but it's the best out there as far as I know of the ones that actually exist. Many years ago I ran into an unsolvable bug with Upstart and a particular complex setup that I couldn't even minimize into a useful bug report. I've had problems with systemd, same as I've had problems with, say, Linux and glibc, but they've all been quite solvable.)

[jandrese]

When Systemd is working it is a lot more regular than the old init systems. You have nice tools like nmcli that combine the functionality of literally dozens of programs into a single interface.

The problem comes when you discover that the Systemd version of a service is missing one crucial feature you need or is buggy, so you need to run an external service instead and it starts fighting you tooth and nail. Or when something breaks and tracking down the issue is hindered by it being hidden in behind a bunch of layers of indirection.

For example, try configuring an Ubuntu 16 box to randomize its MAC address before searching for WiFi access points.

[qplex]

>randomize its MAC address before searching for WiFi access points

Indeed, things like start order and depedencies can be completely broken on systemd. Many problems arise from incompatibilities with legacy systems and interfaces.

I had a big headache last time I had to configure systemd to wait for dhcp before running other stuff because the systemd targets for networking were not working - part of the problem was a legacy interface for network configuration.

[kasabali]

nmcli is a NetworkManager command.

[e12e]

I recently had to write/adapt an init.d script for caddy2 on a legacy server. I must admit even such a simple task, did indeed feel a little bit arduous - I suppose I can relate better with those railing against init-scripts, now that it's been some years since I last had to touch one... (we needed something with a recent ssl implementation to terminate ssl on that box, caddy fit the bill - serving as a stop gap..).

That said, it should be eminently possible to have something like service files, and not systemd.

And I must admit, reading the back and forth on the linked issue, I still have zero confidence in systemd upstream as maintainers of systemd.

That, more than the disturbing complexity, "eat-the-world"-tendency and tight coupling (with lip service to separation, by having tightly coupled parts of the overall monolith be "separate" as individual binaries) worries me most about systemd.

https://github.com/systemd/systemd/issues/13674

[jka]

Thanks for highlighting the GitHub discussion of the issue.

I'd imagine the maintainers are extremely busy and may have a sense (or incentive?) to respond - in any way - to issues swiftly, and that closing them may in many cases be the most time-efficient way to keep their own work under control.

That said I agree that situations like this are frustrating for users, especially when they expose what appear to be genuine and reproducible issues.

[e12e]

I get that they're busy, and "lol, you're on an ancient version #wontfix" is quite all right as a first response (maybe a little less so for RedHat subscribers, considering the link between systemd and RedHat).

But not seing that the design seems fundamentally broken, that's more of an issue (message amplification to millions of messages, the fact that it's questionable if systemd even can use the information for meaningful ordering/dependency reseloution.... When it's not systemd doing the mounting..).

[rhinoceraptor]

One thing that drives me insane on my home server is I have a couple service files symlinked into /etc/systemd/system, and about once every other reboot, systemd will forget about those services and not start them.

[cp9]

I only have good things to say about systemd. it's much easier to reason about dependencies, and timers always fire when they should. It's also much easier to debug why a timer isn't working because you can `systemctl run my-job.service` and run that job in exactly the same environment it'll run in with the timer. Yes, it's two files to describe the job instead of just crontab but it comes with a ton of advantages

[znpy]

I have. My machines boot reliabily, my unit files work almost unmodified across my centos machines and my Ubuntu laptops.

I no longer have to learn a three-dimensional grid of init systems (init system, distro, distro release).

If this isn't a positive experience...

Systemd requires a small initial study effort in order to be understood.

If you're unwilling to do that kind of things (is: reading the fine manual), you might as well go buy a macbook.

[pmlnr]

Story from yesterday: I foolishly modified fstab, like I did for over a decade, to swap a mountpoint from a local drive to NFS. What happened: `mount -a` ran fine. No errors. Except... the mountpoint was empty, there was no match in `mount` output to the line.

After poking the internet:

    After altering fstab one should either run systemctl 
    daemon-reload (this makes systemd to reparse /etc/fstab 
    and pick up the changes) or reboot[^1].

So, the summary with systemd: it will break something for everyone sooner or later. It hopefully won't be something you can't fix, but it will make you very angry and in immediate need of a tea break.

BTW, this is not the first one for me, but certainly one of the most frustrating ones, and not because of the behaviour: because of the lack of error messages on stderr or at least in syslog. The messages in ~~syslog~~ journald were casual.

[^1]: https://unix.stackexchange.com/questions/169909/systemd-keep...

[znpy]

Actually it's been like that for a long time, and for a good reason.

/etc/fstab is parsed by systemd and converted int .mount unit files, so that they can be mounted when a) asked for it (auto/noauto) and b) as soon as possible.

think of the _netdev flag: it signals systemd that a mountpoint is network-based, and thus it will attempt to mount it only after the networking target (instead of waiting for it to timeout at boot time, fail , and possibly letting processes start reading and writing from/to an empty directory).

Running daemon-reload is generally system(d)-wide though: whenever systemd configuration is altered (think adding/removing/updating an unit file) you should let systemd know about that by running systemd daemon-reload (think of apachectl reload).

[mcdee]

It's non-obvious, though. Change something in /etc/systemd you can expect to have to tell systemd about it. Outside of that directory, there's no indicator you should do so.

"Long time" is also relative. I've been updating /etc/fstab directly for over 25 years.

[pmlnr]

> and for a good reason

Yeah... there's always a "good reason" with systemd, isn't there?

Still doesn't account for the lack of error message.

[qplex]

It's kind of funny how systemd breaks often when old farts (like us) tinker with something in a way we shouldnt.

[michaelmrose]

Why doesn't it reparse fstab before executing mount commands?

[znpy]

the mount command is not part of systemd

[michaelmrose]

mount the command DOES read /etc/fstab I believe the discussion was a systemd mount unit

[conradfr]

I tried to use LXD containers on a fresh Ubuntu 18.04 yesterday and between those two and systemd it was not really working.

In the end I gave up. I can't put the blame on one part specifically but it was not a good experience.

Other than that I don't see much difference on defining a service with upstart or with systemd, for my basic use.

[mercora]

i used to think like this but honestly i am somewhat convinced of systemd by now. Whlie it feels like a completely different approach for no apparent reasons, the more i have to handle systemd and "the others" the more i prefer the way its done in systemd.

I don't like the feature creep though.. we have now systemd, systemd-udevd, systemd-journald, systemd-networkd, systemd-timesyncd, systemd-resolved, systemd-homed and who knows what else. worse some of these are kind of irreplaceable in the systemd ecosystem.