[jaryd]

Would be interested to hear from the devs about the actual reasons they chose to write unilog (https://github.com/stripe/unilog) instead of using multilog (which ships with Daemontools).

Agreed that multilog is arcane and weird but what are the actual problems with it that unilog solves?

[ebroder]

(I'm an infrastructure engineer at Stripe)

This actually took a fair amount of digging! We've been using some version of unilog for over 4 years now (longer than I've been at Stripe), and we'd mostly forgotten why we switched. What follows is more the result of historical exploration and guesswork than authoritative statement of original truth.

I'm fairly confident that the impetus for unilog was timestamp prefixes for our log lines. We wanted timestamps (so that we weren't dependent on all applications adding them). multilog is capable of doing writing out timestamps, but it formats them with TAI64N. We wanted something more human-parseable.

Once we had it, we started adding other features. These days, I'd say the most useful thing unilog does for us is buffer log lines in memory. We would occasionally see disk writes on EC2 hang for long enough that the in-kernel (64k) pipe buffer would fill up and cause applications to stall.

[ebroder]

And an update! I talked with the engineer that wrote unilog originally.

The original headline feature of unilog was that it wouldn't block writes if the disk filled up. multilog does - if it can't write a line to disk, it stops ingesting data off of stdin, which eventually causes the application to hang writing to stdout.

unilog sends you an email and starts dropping log lines, which we decided better matched the tradeoffs we wanted to make - losing logs sucks, but not as much as blocking your application until you figure out how to free up disk.

[jaryd]

Wow--thanks for the update. Buffering to memory before writing to disk is definitely a great feature (especially when operating out of AWS) and could be enough for us to switch to unilog.

Regarding the timestamps. We've actually gotten quite used to TAI64N--it's definitely not human parseable but it is extremely specific :) We end up just piping logs through tai64nlocal or converting it at the logstash stage.

[pablasso]

Thanks for the follow up!