We log TBs per hour and grep is enough for me to find interesting data quite effectively.
The problem with weird log formats is recreating all the neat stuff you can do with tooling not necessarily just being able to open a file in a text editor.
I think this is a really good point. A logging system could theoretically toggle "text" mode on and off, giving human readable logs in development and small scale deployments.
> In fact, I'm going to build a toy one in python!
I suggest building it as a normal python logging handler instead of totally custom, that way you don't need a "text" toggle and it can be used without changing any existing standard python logging code. Only requires one tweak to the idea: Rather than a template table at the start of the file, have two types of log entries and add each template the first time it's used.
Drawback is having to parse the whole file to find all the templates, but you could also do something like putting the templates into a separate file to avoid that...
I am writing code on my machine, running one query at a time. I can easily view the logs and spend a lot of time looking at them.
I am running test suites, running thousands of queries. It's harder, but I will still view the logs around failures.
Then I am taking the very code, and pushing it to prod. Should my logs be suddenly completely different in this case?
(The right answer of course is to have "log-to-text formatter", either running in-process or as a separate post-processing step. But it better produce nice-looking, human-readable logs, or every format will be "Message: %s" )
The problem with weird log formats is recreating all the neat stuff you can do with tooling not necessarily just being able to open a file in a text editor.