[ledauphin]

One way to reframe this is: "as a user [of the logs], what might I want to know?"

In my experience, this post is often right (and the logs are often wrong). There's a tendency to either log too much or log too little - if only a few items are getting processed, it's fine and maybe even good to log all 7 of them.

But if many, many are getting processed - you'll experience semantic overload as a reader of the logs. What you want is a compressed form

Logging per time interval can be a very handy approach. In my work, we've settled on a hybrid approach - calculate in real time how often things are happening and then log the number of things that have happened, but at a rate that is roughly one log every N seconds.

This takes some more engineering up front but is remarkably often what a log reader actually wants.

[kiitos]

The "compressed form" of logs you're describing here is really just metrics...

[ledauphin]

it's not, though. sometimes you need actual active logs to tell you that something is progressing.

[kiitos]

You don't need logs for that, you can get it from metrics just fine. Via absolute value counters, or some kind of percentage-complete gauge, either way you can easily see progress over time...