[marcosdumay]

So, the observability community is finally so full of ungrounded astronauts that "log" isn't the place you go to look for errors anymore and it's so full of noise that if you push the errors there, you will lose them.

Well, they've going that way for ages. But officially entering the stage where it's better to always ignore whatever they have to say is an important landmark anyway.

[ahofmann]

I'm doing both. A line of logging that looks like "[date without timezone] Foo: bar additional stuff, almost no context" so, almost all lines found in /var/log/*/*.log, are mostly useless.

Right now, I'm playing with the idea to log json in those files and send this also to Signoz, Sentry, graylog, or whatever. This way, my logfiles are still tail-able and grep-able. Very important for development and "the customer is furious, WTF is going on in production" situations.

And also some other software can email me, according to some rules. This helps to have less furious customers. And maybe it can draw some cute graphs, so I can tell management that prod runs great!

[marcosdumay]

If you need an activity log as complementary information, you can send it into another stream. You can have more than 1.

Logs should be readable. What that means you should get the relevant information by reading the file after maybe searching for something a few times. That requires that the information there should be about a single topic. If that's too much, it should be tagged so that your tooling can make it readable, but this is for huge services.