[vidarh]

> The init system runs as PID 1 and strictly speaking, the sole responsibility is to daemonize, reap its children, set the session and process group IDs, and optionally exec the process manager.

The process manager gets killed. How do you recover?

If you have respawn logic for it in PID 1, how do you log information about a failure to respawn the process manager?

Perhaps you build in some basic logic for logging. Where do you store the data? What if the user level syslog the user wants you to feed data to can't be brought up yet, because it depends on a file system that is not yet mounted?

There may very well be alternatives to the systemd design, but I've yet to see any that are remotely convincing, in that most of them fail to recognise substantial aspects of why systemd was designed the way it is, and just tear out stuff without proper consideration of the implications.

Most proposed alternative stacks to systemd falls down on the very first question above.

I agree with you that it doesn't seem like a great idea to stuff everything in the init system, but I don't agree that "one must delegate" unless the delegation reduces complexity, and I've not seen any convincing demonstrations that it does.

I'd love it if someone came up with something that provided the capabilities and guarantees that systemd does with indepenent, less coupled component, though.

But there's no way I'm giving up on the capabilities systemd are providing again.

[lmm]

Wait what? What happens if the process manager crashes if you're running non-systemd: you might respawn it but not be able to log the fact that you did so. Worst case, you fail to respawn it and your system crashes.

What happens if the process manager crashes if you're running systemd: the process manager is in PID1 (or, equivalently, in a tightly coupled process that PID1 depends on - because the whole point of your post was that you can never get to a state where PID1 is working but logging isn't working), so your system crashes, every time. How is that better? And if that's really what you want, it's easy to configure a decoupled init system to do that.

Hey, some people like their logs to be sent as email. Maybe we should move sendmail into PID1 as well.

[edwintorok]

I think the ideas presented in the 's6' init system address most of these issues, I don't know why none of the distributions picked it up as an alternative: http://skarnet.org/software/s6/why.html http://skarnet.org/software/s6/s6-svscan-1.html

[vidarh]

It doesn't address the logging issue, as far as I can tell. It appears to rely on the same logging solution as the original daemontools. I used daemontools extensively for a while, and it was great, and I like Bernsteins design philosophy which appears to have been largely carried forwards into s6, but it was simplistic, and suffers from a number of the same problems as a "raw" SysV-init, such as putting us back at the mercy of badly written start/stop scripts, and no dependency management.

If someone could come up with a systemd replacement which manages to keep the systemd features while using a design philosophy more in line with that of Daemontools, that would be fantastic, but it'd end up looking very different to s6. Some stuff could certainly be cleanly layered on top (such as using a wrapper to avoid the start/stop problem using the same method of cgroup containment as systemd). Other things, such as explicit or implicit (via socket activation etc.) dependency management, I'm not so how you'd fit into that model easily.

I'd love it if someone tried, though. It would certainly make it easier to experiment with replacing specific subsets of functionality.

[JdeBP]

> If someone could come up with a systemd replacement which manages to keep the systemd features while using a design philosophy more in line with that of Daemontools, that would be ...

... in the indicative rather than in the subjunctive, and in fact already mentioned here once. http://homepage.ntlworld.com./jonathan.deboynepollard/Softwa...

> The process manager gets killed. How do you recover?

In nosh terminology, this is the service manager. If it gets killed, the thing that spawned it starts another copy. This could be systemd, if one were running the service manager under systemd. It could be the nosh system manager. Of course, recovery is imperfect. If one designs a system like the nosh package, one makes an engineering tradeoff in the design; the same as one does when one designs a package like systemd. The system manager and the service manager are separate, but the underlying operating system kernel will re-parent orphaned service daemon processes if the service manager dies. One trades the risk of that for the greater separation of the twain, and greater simplicity of the twain. The program that one runs as process #1 is a lot simpler, being concerned only with system state, but there's no recovery in a very rare failure mode. Indeed, the simplicity makes that rarity even greater, if anything. systemd makes the tradeoff differently: there's recovery in a very rare failure mode (which I've yet to see occur in either system outwith me, with superuser privileges, sending signals by hand) at the expense of all of the logic for tracking service states, and for trying to recover them (in circumstances where one knows that the process has failed somehow and might possess corrupted service tracking data), all in that one program that runs as process #1.

> If you have respawn logic for it in PID 1, how do you log information about a failure to respawn the process manager?

In the log that is there for the system manager. See the manual page for system-manager, which explains the details of the (comparatively) small log directory and the (one) logging daemon that is directly controlled by the system-manager, both intended to be dedicated to logging only the stuff that is directly from the system manager and service manager.

> Perhaps you build in some basic logic for logging. Where do you store the data?

In a tmpfs, just like systemd-journald does in the same situation. /run/system-manager/log/ in this particular case. Strictly speaking, this "basic logging" isn't built-in. In theory, it is replaceable with whatever logging program one likes, as the system-manager just spawns a child process running cyclog and that name could be fairly simply made configurable. In practice, difficulties with the C++ runtime library on BSDs being placed on the /usr volume rather than the / volume, and indeed the cyclog program itself living on the /usr volume when it has to be under /usr/local/, have made it necessary to couple more tightly than wanted here, so far. But those problems could go away in the future; if the BSD people were persuaded to put the C++ runtime library in the same place as the C runtime library, for example.

> Most proposed alternative stacks to systemd falls down on the very first question above.

In many ways, that's because it's a poor question that focusses on a very rare circumstance. As I said, I've yet to see either system exhibit this failure mode in real-world use absent my deliberately triggering it. (Nor indeed have I ever seen it occur with upstart or launchd.) Much better questions are ones like "Where are inter-service dependencies and start/stop orderings recorded?", "Is there an XML parser in the program for process #1?", "What makes up a service bundle?", "How do system startup and shutdown operate?", "How does the system cope with service bundles that are on the /var volume when /var hasn't been mounted yet?", "How does the system handle service bundles in /etc when the / volume is read-only?", and "What does the system manager do?". Those are all answered in the package's manual pages and Guide, of course.

[dalias]

People actually _want_ the logging behavior of systemd? My impression is that it's the most widely hated part; I've heard endless stories of journald thrashing the filesystem forever, losing logs completely on corruption, etc. And even operating properly, its performance is comparable to grepping a flat text log, since despite having a "more efficient" format, it increased the actual data size by something like 4-10x.

Logs are essentially write-once, write-often, read-rarely data. As such, the optimal format is always going to be a flat, append-only file.

[edwintorok]

Also coredumps don't really belong into the journal, I'd turn that off.

[hueving]

Isn't that a pretty narrow corner case? I can count the number of times the process manager has been killed on one hand.

[vidarh]

Add enough machines, and "narrow corner cases" happens all the time and at all the wrong moments.

The bigger point is that there are lots of these "narrow corner cases" all over a typical SysV-init setup, not least due to tons of badly written init scripts. The number of times services have failed to start

To produce a systemd alternative, creating something that competes favorably with SysV-init is insufficient. Today you also need to demonstrate how you deal with those corner cases, or why they don't matter - many of us have no intention of going back to the bad old days.

[edwintorok]

Also you depend every-day on another process that is special in some sense just as the process manager: Xorg. If Xorg dies all your desktop applications die. By your line of reasoning Xorg should be moved into PID 1 too, which is definetely not a good idea.

I don't say that Xorg hasn't crashed, it did rarely when running RC code or proprietary drivers. In fact I probably had as many Xorg crashes as kernel panics, which says something about how stable Xorg is. Still I wouldn't want to run it as PID1, where a crash would really bring down everything.

[vidarh]

I don't understand how you come to the conclusion that putting Xorg in pid 1 would be even a remotely fitting comparison.

For starters, as an example, I have 100 times as many servers than I have desktops to deal with - for a lot of us Xorg is not an important factor. But the process manager is vital to all of them - server and desktop alike if you want to keep them running. If the process manager fails, it doesn't matter if it wasn't Xorg that took things down.

Secondly, that X clients fail if the server fails is not a good argument for moving Xorg into pid 1 too, because it would not solve anything. If pid 1 crashes, you're out of luck - the best case fallback is to try to trigger a reboot.

Having (at least minimal) process management in pid 1 on the other hand serves the specific purpose of always retaining the ability to respawn required services - including X if needed. (Note that it is certainly not necessary to have as complicated respawn capabilities in pid 1 as Systemd does).

Having Xorg in pid 1 would not serve a comparable purpose at all: if it crashes, the process manager can respawn Xorg. If you then need to respawn X clients, and be able to recover from an Xorg crash, there are a number of ways to achieve that which can work fine as long as your process manager survives, including running the clients under a process manager, and have them interface with X via a solution like Xpra, or write an Xlib replacement to do state tracking in the client and allow for reconnects to the X server.

Desktop recoverability is also a lot less important for most people: Every one of our desktops have a human in front of it when it needs to be usable. Most of them are also rebooted regularly in "controlled" ways. Most applications running on them get restarted regularly. People see my usage as a bit weird when I keep my terminals and browsers open for a month or two at a time.

On the other hand, our servers are in separate data centres and need to be availably 24x7, and many have not been rebooted for years, and outside of Android and various embedded systems, this is where you find most Linux installs.

While we can remote reboot or power cycle most of them, with enough machines there is a substantial likelihood of complications if you reboot or shudder power cycle (last time we lost power to a rack, we lost 8 drives when it was restarted. Even with "just" reboots there is a substantial chance of problems that requires manual intervention to get the server functional again (disk checks running into problems; human error the last time something was updated etc.)

That makes it a big deal to increase the odds of the machines being resilient against becoming totally non-responsive.

[edwintorok]

I think you raised an interesting point here 'for a lot of us Xorg is not an important factor', I agree. The same could be said about some of the features that systemd provides that cause a lot of flames (binary logs). It has been said before that systemd is monolithic, and this is probably what makes switching so hard.

It is all-or-nothing, whereas if you could gradually replace the old sysvinit/policykit/consolekit/etc. stuff with systemd/logind then problems during that transition could be debugged more easily. You could also choose to not replace some components where the systemd/non-systemd replacement is broken.

[andreasvc]

That is a pretty bizarre argument. I would conclude from init and Xorg rarely crashing that it is possible to write a reasonably stable daemon, and that perhaps it's not a good trade-off to introduce a lot of complexity into those daemons to be able to recover from crashes.