[loxias]

I don't like the design at a fundamental level. It feels brittle, bloated, and not very unixy. I'd rather have my init system be a handful of microscopic executables, using text files or symlinks for configuration and text files for logging.

I don't like everything depending on systemd it feels like too much complexity at the wrong part of the stack. It doesn't jive with my sense of architecture, it's not well designed software.

[derefr]

Systemd does use "text files or symlinks for configuration" — those being the unit files in /lib/systemd and /etc/systemd. `systemctl enable` just makes a symlink from /lib/systemd into /etc/systemd, even. What would you point to to claim that it does otherwise?

> text files for logging

...just sucks, on both embedded systems and production servers. (I.e. anywhere where you aren't debugging the machine on the machine, but rather from another machine.)

Either the program just writes to a plain text file forever — and so fills up your disk the first time it goes haywire (so now you have two critical runtime problems!); or it implements its own log rotation and compression (as must every other daemon — not very unixy!); or it must be specifically wired to work with syslog APIs in order to use rsyslog (which, by the way, uses binary wire protocols as well; logging at scale hasn't been text-based in a long time.)

Journald, meanwhile, just sits on the other side of the pipe from any systemd service-unit's stdout + stderr; manages log rotation + compression in a centralized way (which also means you get cross-unit log compression for free); and offers CLI tooling to pipe the multiplexed log stream back into anything that wants to read from it, in whatever format those things want to read from it (i.e. tools that want JSON Lines, get JSON Lines; tools that want plaintext, get plaintext; tools that want a binary record stream, get a binary record stream.)

Is this a Unixy approach? Well, it's pretty much the same one taken by the extremely venerable Unix/Linux line-printer (lp) subsystem — CLI commands, with textual config files, for interacting with a system daemon (lpd) that manages and manipulates binary state files, within daemon-owned directories. Would you complain that the contents of /var/spool/lpd aren't human-readable?

[bayindirh]

> (Text files for logging) ...just sucks, on both embedded systems and production servers.

Ehrm, no. Managing a sizeable fleet, with a central logging server for 1.5 decades, and we never had the problems you mentioned:

> and so fills up your disk the first time it goes haywire.

This is a bug of the program or configuration mistake or your monitoring is not working as intended.

Funnily, we're seeing more disk pressure from systemd journals. Go, figure.

Just remembered: syslog daemons have rate-suppression mechanisms to prevent big lines repeating too fast and preventing your disk from filling up. So even your program enters an infinite loop, a well configured syslog daemon (rsyslog, syslog-ng, whatnot), should note "X similar errors have been supressed.", where X can be anything from 2 to 1000 (or even more).

> or it implements its own log rotation and compression

Which you can disable 99% of the time and just delegate the stuff to logrotate.

> it must be specifically wired to work with syslog APIs in order to use rsyslog

rsyslog is just a syslog daemon. syslog is kernel plumbing at this level. You can terminate this pipe with anything.

> Journald, meanwhile, just sits on the other side of the pipe from any systemd service-unit's stdout + stderr; manages log rotation...

And provides nothing new when compared to syslog plumbing. A binary log, some tooling around that, and that's it. It even makes per daemon log monitoring harder by blinding syslog-aware monitoring and automation tools, hence we need to enable rsyslog on the system too. Now we have two journals. Neat.

> venerable Unix/Linux line-printer (lp) subsystem

Which only handles "line-printer" subsystem, and yes, it's more UNIXy. It doesn't get the text output, bashes into a binary data structure, and doesn't try to replace anything and everything from boot to logs to time sync to user login to tap water temperature.

It just stores its state in binary file. Which almost every UNIX daemon does. Incl, but not limited to X11 & CUPS.

[duskwuff]

> Funnily, we're seeing more disk pressure from systemd journals.

So configure journald correctly? It has multiple options to control disk usage from logs -- `man journald.conf` and search for "MaxUse" for the relevant options.

> Just remembered: syslog daemons have rate-suppression mechanisms

So does journald. Relevant options are RateLimitIntervalSec and RateLimitBurst, and individual services can set their own limits as well.

[yjftsjthsd-h]

> So configure journald correctly? It has multiple options to control disk usage from logs -- `man journald.conf` and search for "MaxUse" for the relevant options.

So configure logrotate correctly; you hardly need journald for that.

[amluto]

Systemd had no shortage of issues, but:

> text files or symlinks for configuration and text files for logging.

Systemd gets this right and arguably pushed the whole ecosystem in this direction. The old rc scripts could barely be considered a text file and symlink configuration system — they were a pile of text files containing a miserable combination of code and configuration mixed together, along with a very simple configuration (this service is enabled in these runlevels, more or less) that got translated, hopefully correctly, into symlinks. Of course, nothing really kept the symlink farm consistent with itself or anything else except a pile of additional scripts associated with packages that tried and usually succeeded.

[loxias]

I agree with (basically) all of this, the larger point. I'll note that I said "or"... ;-) (CYA)

I'm not sure what the "right" solution looks like, perhaps a directory of TOML or JSON files. Perhaps the aforementioned + executable shell scripts with predictable naming? handwave handwave as long as it's "UNIXy". (consist of easy to edit text & not invent a new anything & be composed of pieces which do one thing well)

[hyperpl]

Do you think we'll get lucky with a systemd replacement similar to how pulseaudio has been deprecated by a much more reasonable implementation Pipewire?

[amluto]

In my copious free time, I have a vague idea of designing and implementing a mechanism called kpid1.

Basically, a task running as pid 1 (including in a container) could call a new kpid1() syscall, which would cause the kernel to completely take it over. The kernel would take care of all the usual init work, and it would expose the minimal API (presumably using a new kind of fd) to allow a different task to give it instructions and manage zombies as needed. And that’s it.

It’s worth noting that the entire concept of pid 1 is very unixy, but not in a good way. Reasonable modern designs (e.g. all the Windows variants) don’t have any real equivalent.

[yjftsjthsd-h]

What benefit are you seeing in putting it in the kernel?

[amluto]

Several:

Zombie reaping could have a reasonable API. (Signals are miserable.)

PID 1 is magic in problematic ways. In particular, if PID 1 crashes, the whole system goes down with it. And having PID 1 be a normal program running from a normal ELF file means that that ELF file is pinned for the life of the system or at least until it execs something else. So handoff from initramfs to a real fs either involves PID 1 calling execve() or involves leaving the init process around. Upgrading the package containing PID 1 requires execve(). Running PID 1 from a network filesystem or an unreliable device risks a kernel panic for no good reason.

With PID 1 moved to the kernel, the actual service management job is no longer coupled to PID 1’s legacy. A service manager could hand off to another one by saving its state to disk and exiting, by running the new one and moving its state after the new one starts, or by any other ordinary means. And if it crashes, you can ssh in, read logs, save work, and then restart it or the whole system as appropriate.

As a minor additional benefit, having PID 1 in the kernel could enable some optimizations. Right now, a process must enter the zombie state when it exits, and it must stay in that state until its parent wakes up and reaps it. So a service exiting fundamentally involves some complex bookkeeping and a context switch to a single, unrelated process. If the kernel knew that kpid1 was in use and that nothing in the system actually needs to be notified of exiting children of pid 1, then a child of pid 1 that exits could simply go away, as it would on a sensible system like Windows.

(Yes, it's okay to admit that, in some respects, Windows is substantially better than Linux/Unix.)

[MarkSweep]

Not OP, but the whole business of PID 1 having to reap orphan PIDs seems like something the kernel should have to do. Is there a good reason for when a process exits that not other process is waiting on, that a user-mode PID 1 process has to observe that exit?

[yjftsjthsd-h]

My understanding is that reaping children is a normal thing for most processes to do, and it's only orphans that fall through to PID 1, at which point it's easier to deal with it there rather than need to do anything special in ring zero.

[amluto]

Reaping children is "normal" in a universe where processes have numeric ids that can't be reused for unrelated processes until some handshake occurs that frees the id for reuse.

If you take anything resembling a fresh look at this concept, it's absurd. Imagine if every open file had a systemwide unique id, and one specific process owned that id and would continue to own it until it released it.

Reasonable designs use weak references that don't have values that can be compared across processes. These are usually called "handles" or "file descriptors", and they don't have this problem at all. Nothing reaps sockets, for example, and nothing needs to.

[warrenm]

I have to think ... you don't actually know what you're talking about

I fought against systemd for a while, too

I was wrong

The "if it ain't broke, why fix it" approach with 'classic' init scripts led to a far far messier place than systemd

My only complaint about systemd is that I haven't found a way to push the journal to text simply ...and that's most likely my poor google-fu or not having enough to time to fully dive into it, rather than "systemd's journal sucks"

Systemd does use "text files or symlinks for configuration"

It's very well designed - though you may happen to have a preference for a different architecture (...but the one you described is pretty much systemd, my friend :))

[jerryzh]

Hear your words.

How is that different from I don't like relativity theory at a fundamental level. It feels abundant, bloated, and not very physical. I'd rather have my Newton's law with a handful of augmentation.

[kaba0]

Did you consider in your opinion that booting and service management is an insanely hard problem?

Would you write an optimizing compiler in multiple small tools as well? Essential complexity can’t be reduced, IPC will blow up your accidental complexity budget. Many times a monolith is indeed the best design choice.

[paulddraper]

> text files or symlinks for configuration

Like...Systemd?