[gmmeyer]

I have never encountered the need for this and I honestly doubt you have either. Systemd fails mysteriously far less often than poorly written init files I have seen and it is honestly not that hard to debug almost every failure I've ever encountered. And I have worked on some very large scale systems with systemd. A systemd file I helped write has run many millions of times over without issue. I honestly find this comment impossible to believe

[mcspiff]

I’m a big fan of systemd, but we’ve definitely run into actual bugs in it. Especially the early RHEL7 days (7.0-7.3). Some were fixed upstream, some we’ve worked around, etc.

[WesolyKubeczek]

In the beginning, the road was quite bumpy, and my irk was that people's use cases were being dismissed in a very cavalier fashion: like, if I didn't think of a workflow, or a use case, or a piece of software that doesn't play nice which you cannot change, it's not valid and let it burn. Oh, and whoever points these things out, or ridicules me for this attitude, is a troll or a hater.

In 10 years, though, reading recent bug reports, I can see that the project's leads have grown out of it, largely.

Case in point, they had recently put a bug into a 24x release that made a lot of machines, including mine, unbootable, but the fix was just as quick.

[InvertedRhodium]

There is still no way to filter application output at collection time, which can functionally make journald useless if you have an application that is too chatty, and will likely never be given that (AFAICT) there is hefty ideological opposition from the systems developers¹.

¹https://github.com/systemd/systemd/issues/6432

[WesolyKubeczek]

Isn't this something that, to a degree, might be mitigated by rate limiting journald has?

IMO Lennart is right in that collection should be optimized to a point where it is (almost) never the bottleneck. I, personally, would put a filtering program between the chatty application and journald.

[InvertedRhodium]

The scenario I personally have is:

- Legacy application, with no access to source code.

- Company that wrote the application hasn't existed for a decade.

- It logs several useless lines every second. This is non-configurable, and includes a timestamp which makes each linue distinct.

- It also logs some very important lines that _must_ be responded to.

I don't think this is a unique situation to find yourself in, and currently this application is permitted - explicitly - to spam the journal to a point that it is effectively useless. Yes, you can use a filtering application to sit in between systemd but that's a workaround and not an actual solution (as stated in the issue linked above):

- it could not be "plugged in" via overlay (i.e. modification of Exec... is needed - this gets ugly especially if you have ExecStartPre/ExecStopPost)

- SyslogIdentifier has to be set to produce proper name in journald

- The filter itself has to be quite reliable (to not hang) and a bit sophisticated to properly handle signals, process shutdown, low disk space etc.

[WesolyKubeczek]

What you need is a pipe through grep. It’s as reliable as it gets. I don’t understand, though, why your log filter should all of a sudden grow disk space monitoring and signal handling. This is what systemd and journald should be doing, and in their recent versions, they are quite capable.

[SiberianDweller]

How about to open this exceedly verbose legacy program with IDA and fill with NOPs all the places where this program logs something useless? I mean, this is an activity which really should be named as "hacking", in opposition to playing with init scripts.

[mcspiff]

I generally agree, but will say that’s not always easy if it’s something system level that’s being induced to be too chatty —- you’re stuck replacing the upstream unit’s Exec, which frankly kind of sucks. I’m also trusting the filter to be as reliable, which may or may not be true.

Rate limiting works, but it’s a pretty blunt instrument and can lead to losing your actually valuable entries vs capturing garbage.

[gmmeyer]

To be clear it definitely used to be more buggy, but I haven't encountered a genuine bug in a long time

[mcspiff]

Might depend on your environment too, I generally find the newer you can get for systemd the better off you are (which is really quite high praise!). Unfortunately the life cycle for some of our environments is quite long for things like RHEL7 -> RHEL8.

[zh3]

Ever worked on systems where you're on the hook for the guarantee that you can keep it running?

We had 3 major breakages due to systemd, which is why we went back to something pretty much anyone who knows a bit of bash scripting can fix.

[bonzini]

And you didn't have separate test and production sites?