[doix]

Ha, systemd is basically my git. I'm always surprised by how many developers don't grok git despite understanding significantly more complex things. I keep thinking if they would just take a day or two, they would get it and then wouldn't complain about having to copy paste commands and "rm -rf && git clone" to fix their workspaces.

And yet I do the _exact_ same thing with systemd. I don't get it, I don't like it, and every time I need to do _anything_, I'm googling, copy pasting commands into my terminal until something vaguely correct happens. The CLI options are indecipherable to me, like reading hieroglyphics.

Deep down I understand that it solves some pretty complex things when it comes to system initialization, but something about the UX of using journalctl and systemctl is so bad (for me) that it creates a mental block that I cannot overcome.

Luckily, nothing I do in my day job requires messing with systemd.

[js2]

I know how you feel. I finally brewed myself a pot of coffee the other day and just followed the advice I always give everyone else-RTFM:

https://www.freedesktop.org/software/systemd/man/latest/syst...

It's pretty well documented, especially for Linux. Follow the links from that page at least one level deep.

Part of the complexity is that it's tightly integrated into the same Linux concepts that enable containers. In fact, systemd-nspawn can manage containers on your behalf.

But start with the concepts and unit files, then read the man pages for systemctl and journalctl. That'll get you most of the knowledge you need to interact with it on a day-to-day basis.

[michaelcampbell]

> I'm always surprised by how many developers don't grok <things> despite understanding significantly more complex things.

You used git as a (counter?) example, but for me systemd has this issue because it's because I touch it so infrequently that whatever I learned about it has aged out of mental cache.

I have the same issue with `jq`, GNU `parallel`, and `ffmpeg` (which I'll grant even if I used it daily I probably wouldn't be able to keep up.

> UX of using journalctl and systemctl is so bad (for me)

Not JUST you. I've heard all the arguments 100x for "binary logs good/bad", but I feel this was a step down for some puritanical idea that doesn't translate for many to practicality.

[dingnuts]

how on earth is remembering the location of some log file, buried in /var/log somewhere, hopefully, or maybe next to the application in /etc/ or /opt/, easier than remembering journalctl -u unitfilename ? or journalctl -u unitfilename -f to follow the logs -- the very same flag tail uses?

after a decade of systemd usage I've never once had to actually think about the format the logs are stored in, or where they're stored; journalctl is actually easier than using tail and cat

[enriquto]

> how on earth is remembering the location of some log file, buried in /var/log somewhere, hopefully, or maybe next to the application in /etc/ or /opt/, easier than remembering journalctl -u unitfilename ?

This may seem obvious to you, but I for one find filenames much easier to remember. Of course the logs are inside /var/log (and not anywhere else).

Moreover, with filenames, you can check your logs with a program that is independent of the program that produces it. Why implement the -f option on other programs when it is already implemented in tail?

[dingnuts]

That's the thing though, you have to remember filenames. I don't have to remember anything, just journalctl, because it can be run without any arguments and get logs for -everything that's running- including units I totally forgot about!

More than once I've forgotten the name of some unit I wrote -- the same as I easily forget filenames for random services I've installed, and where I might have installed them. I can discover the service name by calling systemctl status, or I can just call journalctl with no arguments and pass directly to grep to find what I need, in whatever unit it's in.

Try that when you have log files scattered all around your filesystem.

[michaelcampbell]

> you have to remember filenames.

You have to remember unit names. They're all just names.

[ecnahc515]

Not really, there’s systemctl list-units.

[lvass]

I concede it's as easy if you're just reading one log from one machine, but what if you're harvesting and/or merging multiple logs from multiple machines? It's much easier to do with simple unix tools and plaintext logs.

[michaelcampbell]

> after a decade of systemd usage I've never once had to actually think about the format the logs are stored in, or where they're stored; journalctl is actually easier than using tail and cat

That's wonderful for you. For me, it is not.

Funny how people are different.

[pdimitar]

This is not a people problem, it's a technical problem.

[marcosdumay]

It's hard to explain the problem with journalctl... it just always does something completely different from what I expect it to do.

The option to fix it is always there, individualized by some rule that is completely different from what a command line interface tends to use, so it's mostly unrecognizable.

It wouldn't really be that bad, except for the fact that it's very hard to be confident it's doing the job you wanted. So it's showing no errors with this service... does that mean the service has no errors?

[takinola]

This Digital Ocean guide [1] is my go-to explainer for journalctl. Digital Ocean has pretty good tutorials all round.

[1] https://www.digitalocean.com/community/tutorials/how-to-use-...

[sph]

Sounds like a you problem. You have identified a hole in your knowledge, you should address it. Systemd is not rocket science, and I would say it's more intuitive that git is anyway.

[sneak]

It’s not rocket science, but it is baroque. TFA ends with 30 lines of config to make it boot and do nothing, including some to disable a bunch of spooky action at a distance invisible default dependencies, which still ends up with a system with a bunch of other units that aren’t explained.

[jambutters]

Richard Hipp (sqlite author) also finds git a pain even though he understands it so he made fossil