[ghostpepper]

Binary log files is another example of something that could have advantages in theory but never seems to work well in practice. What used to be "tail /var/log/nginx.log" is now

man journalctl journalctl -u nginx <G to skip to latest, ctrl+C because it's taking too long> journalctl | grep nginx <wait...> go back to man page to look for more options to search

etc

[traverseda]

One of my earliest experiences with systemd's binary log files was on an embedded system I had ~~chrooted~~ systemd-nspawned into, and every time I tried to get the log output it would segfault (some issue with qemu-static probably). Very frustrating experience.

Really feels like "Embrace, extend, extinguish" but the thing they're extinguishing is unix-as-a-programming-environemnt.

[sliken]

Or a Redhat plan to generate more consulting/support income. Ever wonder why Oracle products were so hard to install/maintain?

[egberts1]

It's the IBM way.

Everything be in 4-digit or 5-digit error code.

Need to RTFM for such codes. Or worse, StackOverflow them.

[yawaramin]

Yes, a for-profit corporation has the nefarious plan to... checks notes ...try to earn more revenues. We certainly can't stand for that!

[sliken]

Heh, well making your product worse for more revenue seems pretty warped, and increases the risk of lost market share. Even more warped is having 2 of your engineers join the Debian technical committee and use the bylaws to force a vote before anyone figures out exactly what SystemD is.

[yawaramin]

Were they coerced to vote for it? Why did they vote for it if they didn't know what it is? Honestly it's hilarious how systemd haters see conspiracies everywhere.

[pikrzyszto]

"tail /var/log/nginx.log" is now "journalctl -fu nginx" or "jouranlctl -eu nginx". Not that much of a difference, it really depends what you grow up with. We could argue "tail" is ugly because you never know if it's "tail /var/log/nginx.log" or "tail /var/log/nginx/nginx.log".

A nice feature is that journald gives you flags like "--boot" to see logs only emitted during a specific boot, or "--since '5m ago'" which is not that straightforward to do with the approach you favor. (if there is a way to do it easily, please let me know!)

[ilyt]

I'd recommend running strace on that journalctl command, you might be surprised what happens. It's even worse if you actually store journalctl logs on disk (most distros defer to storing it in ram). I have it set to store on disk and keep at most 1GB of logs

On my NAS

    # strace journalctl -u nginx 2>&1 |grep open |wc -l
    827

and curiously

    # strace journalctl -u baked-potato 2>&1 |grep open |wc -l
    827

(no i do not have baked-potato service on that machine)

It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful. If the binary database was a SQLite file with columns containing app name/log level/etc. it would actually be useful but current one is just shit.

Bonus round:

    #:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
    #:/var/log/journal time journalctl -u nginx >/dev/null 2>&1

    real 0m3,232s
    user 0m0,036s
    sys 0m0,292s
    #:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
    #:/var/log/journal time ag -R nginx . >/dev/null 2>&1

    real 0m2,531s
    user 0m0,004s
    sys 0m0,290s

slower than actually searching thru files directly...

[pikrzyszto]

Why would it care about stracing? It was never too slow for me. A "bad" performance might be caused by ex. consistency checking. Benchmarks like these don't tell much - you can't reduce software performance to one number.

> It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful.

That sounds interesting, can you please elaborate on the internal structure of journald files or link to further documentation? And why would I care if journald handles it for me?

[ilyt]

> Why would it care about stracing? It was never too slow for me. A "bad" performance might be caused by ex. consistency checking. Benchmarks like these don't tell much - you can't reduce software performance to one number.

Well, I was routinely getting multi-second wait for operations like systemctl status servicename which is not something rare, because it didn't even kept the pointer to file that has last few lines of app's logs. It's not just benchmarks

It also trashes cache with hundreds of MBs of logs instead of stuff apps running on server actually needs. An equivalent of "tail -f" can go thru hundreds of megabytes of binary logs.

[marbu]

I don't have a negative experience with journald file format. That said, the format is documented here:

https://systemd.io/JOURNAL_FILE_FORMAT/

On the other hand I agree that journald seems to call open syscall more often than I would originally expect, which can be a problem for some edge cases[1], but I don't consider this to be a real problem.

[1] https://blog.marbu.eu/posts/2022-11-20-ebpf-journald/

[ilyt]

The problem is:

If cache is hot, you have hundreds of MBs of systemd logs littering your buffer cache, while it could be used for something more useful

If cache is cold simple operations like systemctl status take up to tens of seconds, especially on loaded servers. Example:

    # time systemctl status influxdb >/dev/null

    real 0m11.913s
    user 0m0.013s
    sys 0m0.531s

It doesn't even keep index of "last file where app wrote logs" which causes above.

[marbu]

But this buffer cache usage is caused by readers/clients checking logs for some reason, not logging itself, right? I see no significant difference compared to syslog here: if I grep all logs for something, it will also place these logs into buffer cache.

Your example with dropping cache and running systemctl status is indeed interesting, 11s looks like too much. But it's a number without a context, and I wonder how big a problem this actually is. I haven't noticed it myself before.

> It doesn't even keep index of "last file where app wrote logs" which causes above.

While I definitely see some room for optimization, I'm not sure what you mean here: journald uses one active journald database file, there should not be a problem with figuring out where the file is, should it?

[lmm]

> We could argue "tail" is ugly because you never know if it's "tail /var/log/nginx.log" or "tail /var/log/nginx/nginx.log".

The difference is that you can use ls to find out, the same ls that you use when working with everything else, rather than needing to know some journald-specific thing that will only ever work for journald and will no doubt change again in another 5 years.

[kaba0]

And you can just issue “systemctl” to list every service. Do you blame excavation machines because you can’t drive them when you only know how to drive a car?

[lmm]

If someone switched my car for an excavation machine when the car was working fine I'd definitely complain.

[candiddevmike]

I personally find journalctl -fu $service very cathartic, because I'm almost always typing this command when $service isn't working.

[throwaway894345]

I really need to alias `journalctl -fu` to something because it's always so tedious to type all that out over and over when I'm trying to debug something. Also, is there any autocomplete for systemd unit names? Typing `journalctl -fu amazon-cloudwatch-agent` over and over and over is so much more tedious--ideally it would be something closer to `jctl amaz<tab>`.

[dinosaurdynasty]

There is autocompletion. It works by default on Pop OS and Debian. (You may need to install bash-completion and/or dbus if your install is very minimal.)

[kaba0]

I do get autocomplete for even that with bash (on nixos). Tried to have a look at my config, but I forgot what enables that, but it is a game changer.

[throwaway894345]

Knowing the path to the log file isn't more complex than knowing the name of the systemd unit, but not needing to know the right set of flags to pass because the sane behavior is the default is _really_ nice. That said, I didn't know about `--boot` or `--since`, so maybe I'll hate journalctl a little less (also, I've only ever used systemd-based systems, so I don't think "it's what you grew up on" applies, at least in my case).

[jjav]

> Not that much of a difference

It's a big difference, not because one command, but because tail is part of the standard unix toolset that works with all files. Except now you have this one log file that is special and needs a different unique tool.

[gnaritas99]

No, having a special way to tail your special log file isn't a feature, tail is a general utility used system wide for this for decades; breaking this is idiotic no matter what you grew up with.

[pikrzyszto]

I'm happy we have special-purpose tools for dealing with logfiles. I don't want to craft one-liners for "give me logs around timestamp" or "give me logs about my service's first start after the boot" every other day.

[Brian_K_White]

"I don't want to craft one-liners"

I do. I specifically want that as a more valuable and powerful feature. Undifferentiated, open-ended, general purpose, sharp tools are infinitely more useful.

[rlpb]

Your gripes all seem to boil down to your unfamiliarity, rather than anything specific that stands on its own as a problem. Unfamiliarity is understandable, but if that were enough to avoid progress, we wouldn't never have progress.

> ctrl+C because it's taking too long

Try --since=today. Then you end up in less, and that should be more familiar, including for search. This is then the equivalent of traditional daily rotating log files, with no other knowledge needed apart from --since and --unit (the latter is noted in "systemctl status" output as a reminder).

Then every further feature is an optional bonus.

[ghostpepper]

As other commenters have mentioned, the unfamiliarity is a symptom of the time I spend using less,grep,sed,sort.. etc on a daily basis. If I could use those journalctl flags for everything I do relating to files on the system then I would be very familiar with them.

It's the same rule I use when purchasing kitchen equipment - often it's better to have a single multi-purpose tool than a bunch of specific implements. Otherwise you end up with a drawer full of strawberry-stem removers, pizza slicers, corn-cob peelers, avocado-slicers, etc.

This is to say nothing of the binary format issues that others have discussed.

[jeroenhd]

Maybe nginx logs at usable, but my experience with reading logs in the old way was often "grep servicename syslog" and prayer that the tool didn't have its own magical log file somewhere else.

The backing store may be controversial, but the journalctl command line is much more pleasant. "I want to see log messages from the end of the last boot when the system crashed" is no longer a Rube Goldberg command line involving less/sed/grep/cut/awk and manual scrolling, it's supported out of the box with no configuration necessary.

[Hikikomori]

Can still do that?