[onion2k]

Binary logs are opaque! Just as much as text logs.

I don't agree with the second assertion there. Text logs are only opaque as far as the format is concerned, but not so much as far as the content goes. Using the example in the article;

    127.0.0.1 - - [04/May/2015:16:02:53 +0200] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0"

You can read a lot of information without knowing the format, the application that generated it, or even which file it was in - you know it's something to do with localhost, you know when it happened, you know the protocol, from which you can infer the "304" means Not Modified, and you know it came from a Mozilla agent. That's a lot more information than you could get from a binary log without any tools.

That isn't necessarily an argument against binary logging, but the notion that text log files are opaque in the same way as binary logs isn't really true.

[nosir33]

> That's a lot more information than you could get from a binary log without any tools.

The environment I work in I am frequently looking at logs that other teams generate. If I needed to ramp up on their custom logging toolset just to perform simple queries I am going to give up and waste the the teams time by getting them to perform the queries for me.

[Jedd]

(Background: I'm not a journal apologist. For a fact I'm finding it challenging to make the mental shifts required to become adept with this new suite of system tools on my myriad Debian boxen.)

> That's a lot more information than you could get from a binary log without any tools.

Arguably you need a tool to get the information you showed above - a single line from an apache log. The tool may have been grep, cat, vi, awk, less, or whatever. That it was installed as part of a base-build on your computer, or at the behest of your usual configuration management system, is either kind of aside, or kind of the point.

Journal uses a bunch of diagnostic & query tools that get installed at the same time that the journal is installed. Yes, the tool / command to get the same type of data you're looking at above -- something that is comparably readable to a line from an apache log file -- is going to be different. But only different.

[onli]

It is not only different. It is also less universal.

With a text based logging system, I can take the usb stick with the system that does not boot on my headless homeserver to any computer and read the logs there. I could even boot the original linux system on that server, running a really old kernel and practically no userland tools, and read them there. Cause that server was using journald, that was not possible. Still don't know what went wrong.

[madhouse]

Bringing a few extra tools for forensics should not be a problem. You bring grep, strings, less, and a bunch more to read text logs. Why not bring one more to read the binary dump too?

I'm sorry, but I don't find the "but I can view text on a machine from the last century" argument convincing. We're not in the past century, and when doing forensics, we usually do that on a reasonable machine, where all the tools we need are available. Otherwise its an exercise in futility.

[onli]

> Why not bring one more to read the binary dump too?

For one, because it is not packaged for my distribution. For two, because I get exactly nothing in return. All binary logs do for me is forcing me to use an additional tool.

> I'm sorry, but I don't find the "but I can view text on a machine from the last century" argument convincing.

POGO-E02. I really don't know how old this is, but it has USB-2 and I bought it 2 years ago, though it was marked as classic then. Maybe 2009?

> and when doing forensics, we usually do that on a reasonable machine, where all the tools we need are available

I'm normally doing that at my own environment, with the tools I am used to, and on my machine. Nothing of that includes a binary log viewer.

[Nullabillity]

> For one, because it is not packaged for my distribution.

Are you running Slackware?

[onli]

I wrote that below, that is a Ubuntu 14.04 LTS. The important point is that the theoretical availability - there probably is a PPA somewhere - of journalctl is an additional, unnecessary hurdle.

[seren]

A few years ago, I was called by the manufacturing team to troubleshoot a simulator bench of our own system. The bench had been developed 15 years ago by a subcontractor and was working happily since then. The issue was pressing because it could halt the manufacturing line. I had 0 information or documentation on the bench. All the respective owners had been gone years ago. I was quite happy when I found there was a basic logging system on a serial line.

I think is it more telling about lack of organization rather than wrong technical choice, but sometime you have to deal with legacy systems and it is good to be able to rely on something as universal as text.

[Jedd]

I know it's pretty weak to say so, but that universality is a result of the stage of the migration -- that particular problem will undeniably reduce over time.

But you're right that there are some work flows and use cases where it'll bite you big time. A recent migration to systemd on my Debian lvm-on-dmcrypt laptop caused me some hours of pain, so I'm not unsympathetic.

Back in the early 90's I was involved in managing a very large network of MS-DOS + Windows 3.x machines. The migration to Windows 95 introduced the same concerns, with similar responses. That's the nice thing about working in IT long enough.

[onli]

Thanks for the sympathy ;)

> The migration to Windows 95 introduced the same concerns, with similar responses.

For me, that is the second big large negative point, apart from the missing universal access (which like you said might get better over time, maybe). This route of having a binary journal with its dedicated journal viewers feels awful lot like being on windows. It's the same negative feeling I get when I get in contact with Gnomes regedit clone. Stepping back to Windows 95 is hardly progress.

[Jedd]

Yeah, I may has mis-worded that sentiment. My point is, and I'm not the first to have noticed, that much of IT seems to be profoundly cyclical. Not necessarily bad, other than the implication we don't really learn from the mistakes of each cycle. Compare and contrast the trisolarans.

Anyway, memory may be failing, but the big problem was one of configuration data (typically small volumes) that used to be kept in .ini (text) files, now being shuffled into the registry. There wasn't a size or complexity issue that drove that move, unlike the challenge of managing and merging many large log files from disparate services on multiple hosts.

In the particular case the toolkit did eventually catch up, but it took a very long time (3-5 years for us, I think, to recover the same level of deployment, configuration, automation). With Journal, in contrast, the toolkit's already there, and ultimately I'm just not convinced that 'I don't have Journal tools installed on this computer' is a persuasive argument against the tool.

I'm not saying there are no compelling arguments, just that one isn't.

[paulfurtado]

Could you not view the journal on another system? I'm curious why:

journalctl -D /<mnt>/<other_system>/var/log/journal

wouldn't work in this case

[chronid]

You assume the other system is a linux box with systemd installed. That may be true, or may not be true at all. :)

[onli]

Yes, exactly. That would've worked if my other system would have had journalct. It is a Ubuntu 14.04 LTS, and to my knowledge it does not have a package for that. Even if it had and I just did not know, that is kind of the point why binary journals (and systemd) suck.

[leni536]

It's not like specific part's of journalctl can't be ported to other systems and packaged separately. It's a distribution issue, not a fundamentaly flaw in binary logs.

[chronid]

Sure, the question is why fix something that is not broken? Why should someone write a systemd journal format reader (but journald is just an example here) when text files work already? Why rewrite tools to support it?

If you get to the point when standard unix tools are useless, well, it's time to use a _real_ database and/or log management system. Not the time to write your own. No one is (should be?) going around grepping 100 GBs/day worth of logs.