Hacker News new | ask | show | jobs
by delusional 334 days ago
I've had colleagues try this. It rarely works. Logging every if end up introducing a huge amount of overhead, both in terms of processing power, but especially in terms of storage. You almost always end up having to filter based on some sort of log level that you then turn off by default in production.

The problem with that is that you're now required to reproduce the issue after turning on the logging, and if you already have a reproducer, why not just attach a real debugger?

The overlap of "we can reproduce" but "it has to run on the production server" ends up being practically zero.
2 comments
KronisLV 334 days ago
> why not just attach a real debugger?

Probably due to no direct access to the environments, whereas there being less organizational resistance to toggling logging levels... assuming that the issue is even easily reproducible instead of "Okay, this happened once and cost us X$, it was pretty horrible, we don't know exactly why that happened but fix it before it happens again."

(not an exact case, but a pattern I've seen)

link
haiku2077 334 days ago
Something I've done in the past is send some logs to BigQuery for cheap mass storage and others to Grafana for fast querying and use in live dashboards. Basically a filter rule in our logging agent to send different events to different destinations. I think with some more hacking I could get both datasources into the same Grafana frontend...
link
delusional 333 days ago
The problem you'll run into, if you haven't already, is that you're now just programming against the log stream. The log stream has become part of the operational interface of the application. Changes to logs, and the setup around them, must now be managed with the same care as all the code in production.

The log filter is all of a sudden part of the application, but managed somewhere else. Everybody is now scared of touching the log lines because who knows what filters have been configured. You suddenly have to debug your log setup, and who logs the decisions about the logs that were filtered?

We already have a place to put that logic. It's the application.

link