[jimws]

Who else here dislikes Systemd? I do. There is so much hidden magic in terms of configuration that when things go wrong, they really go wrong. One of the things that burnt me initially was that journalctl logs are not persistent. They disappear on restart. It needs to be configured for persistent storage.

[growse]

> Who else here dislikes Systemd? I do. There is so much hidden magic in terms of configuration that when things go wrong, they really go wrong. One of the things that burnt me initially was that journalctl logs are not persistent. They disappear on restart. It needs to be configured for persistent storage.

The problem I have with the 'hidden magic' argument is that everything that provides some sort of abstraction is hidden magic, until you take the time to learn about it. For more examples of 'hidden magic', look no further than GCC, the kernel, etc.

So then this boils down to "I don't like new things", which is perfectly reasonable. But make that your argument. Sometimes there's a good reason to change stuff, because the old stuff doesn't work very well.

edit as mentioned elsewhere, logs are persisted in /var/log/journal by default. If your distro doesn't create that on install, then that's where your issue is.

[fanf2]

In a traditional unix, when things go wrong you can use common tools like grep and text editors and shell scripts to work out what happened. With systemd, there is no way to lift the lid and find out where it broke and how to fix it. That is why the magic is hidden magic, not learnable magic.

[andrewshadura]

That’s not true: systemd gives you tools to not just lift the lid but actually effectively find what broke.

[growse]

Sure there is. It's just software. All the regular tools work just fine.

I've happily been using grep, strace, etc to debug issues in both systemd and 'traditional Unix' systems for years.

[makomk]

PID 1 is not quite the same as other processes. Historically it wasn't even possible to use gdb or strace on it without patching the kernel, though I think this changed at some point. Any crash in it still causes a kernel panic that brings down the system though.

[VintageCool]

I am bewildered that Storage=Persistent is not the default.

[silon42]

Also, parallel logging to syslog should be enabled always.

[newnewpdro]

> We had to update change /etc/systems/journald.conf with Storage=Auto to make it persistent.

RTFM

       Storage=
           Controls where to store journal data. One of "volatile", "persistent", "auto" and "none". If
           "volatile", journal log data will be stored only in memory, i.e. below the /run/log/journal
           hierarchy (which is created if needed). If "persistent", data will be stored preferably on disk,
           i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to
           /run/log/journal (which is created if needed), during early boot and if the disk is not writable.
           "auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so
           that its existence controls where log data goes.  "none" turns off all storage, all log data
           received will be dropped. Forwarding to other targets, such as the console, the kernel log buffer,
           or a syslog socket will still work however. Defaults to "auto".

> Defaults to "auto".

You just need to have /var/log/journal. This is your distribution/installer failing you, don't blame systemd.

[throwaway8941]

This seems to be a typical case of Ubuntu brain damage. I don't think I've seen this behavior in other distributions (CentOS, Debian, Fedora, even Arch).

>On Ubuntu 16.04, /var/log/journal does not exist by default. Create it to keep and query events from previous boots.

[jimws]

I have seen this in CentOS. Also see https://unix.stackexchange.com/a/159390

[mongol]

OpenSuse is the same

[VintageCool]

Debian 9 and CentOS 7 both fail in this regard.