[pikrzyszto]

"tail /var/log/nginx.log" is now "journalctl -fu nginx" or "jouranlctl -eu nginx". Not that much of a difference, it really depends what you grow up with. We could argue "tail" is ugly because you never know if it's "tail /var/log/nginx.log" or "tail /var/log/nginx/nginx.log".

A nice feature is that journald gives you flags like "--boot" to see logs only emitted during a specific boot, or "--since '5m ago'" which is not that straightforward to do with the approach you favor. (if there is a way to do it easily, please let me know!)

[ilyt]

I'd recommend running strace on that journalctl command, you might be surprised what happens. It's even worse if you actually store journalctl logs on disk (most distros defer to storing it in ram). I have it set to store on disk and keep at most 1GB of logs

On my NAS

    # strace journalctl -u nginx 2>&1 |grep open |wc -l
    827

and curiously

    # strace journalctl -u baked-potato 2>&1 |grep open |wc -l
    827

(no i do not have baked-potato service on that machine)

It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful. If the binary database was a SQLite file with columns containing app name/log level/etc. it would actually be useful but current one is just shit.

Bonus round:

    #:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
    #:/var/log/journal time journalctl -u nginx >/dev/null 2>&1

    real 0m3,232s
    user 0m0,036s
    sys 0m0,292s
    #:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
    #:/var/log/journal time ag -R nginx . >/dev/null 2>&1

    real 0m2,531s
    user 0m0,004s
    sys 0m0,290s

slower than actually searching thru files directly...

[pikrzyszto]

Why would it care about stracing? It was never too slow for me. A "bad" performance might be caused by ex. consistency checking. Benchmarks like these don't tell much - you can't reduce software performance to one number.

> It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful.

That sounds interesting, can you please elaborate on the internal structure of journald files or link to further documentation? And why would I care if journald handles it for me?

[ilyt]

> Why would it care about stracing? It was never too slow for me. A "bad" performance might be caused by ex. consistency checking. Benchmarks like these don't tell much - you can't reduce software performance to one number.

Well, I was routinely getting multi-second wait for operations like systemctl status servicename which is not something rare, because it didn't even kept the pointer to file that has last few lines of app's logs. It's not just benchmarks

It also trashes cache with hundreds of MBs of logs instead of stuff apps running on server actually needs. An equivalent of "tail -f" can go thru hundreds of megabytes of binary logs.

[marbu]

I don't have a negative experience with journald file format. That said, the format is documented here:

https://systemd.io/JOURNAL_FILE_FORMAT/

On the other hand I agree that journald seems to call open syscall more often than I would originally expect, which can be a problem for some edge cases[1], but I don't consider this to be a real problem.

[1] https://blog.marbu.eu/posts/2022-11-20-ebpf-journald/

[ilyt]

The problem is:

If cache is hot, you have hundreds of MBs of systemd logs littering your buffer cache, while it could be used for something more useful

If cache is cold simple operations like systemctl status take up to tens of seconds, especially on loaded servers. Example:

    # time systemctl status influxdb >/dev/null

    real 0m11.913s
    user 0m0.013s
    sys 0m0.531s

It doesn't even keep index of "last file where app wrote logs" which causes above.

[marbu]

But this buffer cache usage is caused by readers/clients checking logs for some reason, not logging itself, right? I see no significant difference compared to syslog here: if I grep all logs for something, it will also place these logs into buffer cache.

Your example with dropping cache and running systemctl status is indeed interesting, 11s looks like too much. But it's a number without a context, and I wonder how big a problem this actually is. I haven't noticed it myself before.

> It doesn't even keep index of "last file where app wrote logs" which causes above.

While I definitely see some room for optimization, I'm not sure what you mean here: journald uses one active journald database file, there should not be a problem with figuring out where the file is, should it?

[lmm]

> We could argue "tail" is ugly because you never know if it's "tail /var/log/nginx.log" or "tail /var/log/nginx/nginx.log".

The difference is that you can use ls to find out, the same ls that you use when working with everything else, rather than needing to know some journald-specific thing that will only ever work for journald and will no doubt change again in another 5 years.

[kaba0]

And you can just issue “systemctl” to list every service. Do you blame excavation machines because you can’t drive them when you only know how to drive a car?

[lmm]

If someone switched my car for an excavation machine when the car was working fine I'd definitely complain.

[candiddevmike]

I personally find journalctl -fu $service very cathartic, because I'm almost always typing this command when $service isn't working.

[throwaway894345]

I really need to alias `journalctl -fu` to something because it's always so tedious to type all that out over and over when I'm trying to debug something. Also, is there any autocomplete for systemd unit names? Typing `journalctl -fu amazon-cloudwatch-agent` over and over and over is so much more tedious--ideally it would be something closer to `jctl amaz<tab>`.

[dinosaurdynasty]

There is autocompletion. It works by default on Pop OS and Debian. (You may need to install bash-completion and/or dbus if your install is very minimal.)

[kaba0]

I do get autocomplete for even that with bash (on nixos). Tried to have a look at my config, but I forgot what enables that, but it is a game changer.

[throwaway894345]

Knowing the path to the log file isn't more complex than knowing the name of the systemd unit, but not needing to know the right set of flags to pass because the sane behavior is the default is _really_ nice. That said, I didn't know about `--boot` or `--since`, so maybe I'll hate journalctl a little less (also, I've only ever used systemd-based systems, so I don't think "it's what you grew up on" applies, at least in my case).

[jjav]

> Not that much of a difference

It's a big difference, not because one command, but because tail is part of the standard unix toolset that works with all files. Except now you have this one log file that is special and needs a different unique tool.

[gnaritas99]

No, having a special way to tail your special log file isn't a feature, tail is a general utility used system wide for this for decades; breaking this is idiotic no matter what you grew up with.

[pikrzyszto]

I'm happy we have special-purpose tools for dealing with logfiles. I don't want to craft one-liners for "give me logs around timestamp" or "give me logs about my service's first start after the boot" every other day.

[Brian_K_White]

"I don't want to craft one-liners"

I do. I specifically want that as a more valuable and powerful feature. Undifferentiated, open-ended, general purpose, sharp tools are infinitely more useful.