[heliostatic]

My biggest quality of life improvement for understanding logs has been lnav (https://lnav.org/) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

[heinrichhartman]

Looks great, but I struggle to get this working with the two main log sources on my system:

1) systemd logs (binary data stored at /var/log/journal)

2) docker logs (from various containers running on the host)

Any hints on getting this working? I don't see this covered in the docs (https://docs.lnav.org/en/latest/formats.html).

[funcDropShadow]

for systemd logs this works for me

    journalctl -o json | lnav

[TeMPOraL]

Thanks for linking to this!

Until now I thought logview.el[0] is the bee's knees, but now I can feel feature envy set in. There are some seriously powerful ideas listed on the lnav page, and it's also the first time I saw SQLite virtual tables used in the wild.

--

[0] - https://github.com/doublep/logview

[francislavoie]

At my current company, we're using pdsh with tail -f piped into sed to follow logs from multiple remote servers at the same time, label them and colorize them. Works okay. Decent solution without needing to install much other software. Not my favourite because it doesn't deal well with SSH sessions timing out and leaves some tail -f processes hanging, and some other quirks. But out of laziness and risk of breaking things in prod, we haven't tried experimenting with much else.

[tstack]

lnav has some support for tailing files on remote hosts -- https://lnav.org/2021/05/03/tailing-remote-files.html

It works by uploading a stub that is an "actually portable executable" to the remote over ssh. lnav then talks to the stub to sync files back to the local host. I don't know the scale of the hosts or files, but if they're not too big, it might work fine.

[UI_at_80x24]

Run that in a tmux session. Problem solved.

[oefrha]

Huh, I almost posted a duplicate recommendation. My only complaint with lnav was that it had to be built from source on Linux and the build was frigging huge. Apparently they have a pre-compiled linux-musl binary now.

[amarshall]

> that it had to be built from source on Linux

Seems to be packaged quite widely https://repology.org/project/lnav/versions

[joshka]

I used nix when I had to make lnav work on some machines where this would have been difficult.

[dj_gitmo]

I love lnav and use it constantly, but it crashes a lot. I do wish there was something like lnav that was a little simpler to use, and written in a more resilient way that crashed less. I can cut lnav some slack for the crashes because identifying and parsing arbitrary log formats seems like a messy problem. Still it shouldn't crash 1/3rd of the time I use it.

[tstack]

Sorry for the crashes :( I've been trying to improve it's internals more than adding features as of late. If you haven't already, please file bugs on github and/or submit crash logs to the mailing list. (I've taken a break from working on it lately. So, if you've done that and I haven't gotten back, I apologize.)

[dj_gitmo]

No worries. It's a great tool and I'd be very happy to make reports and share crash logs.

[chrisweekly]

Yes! lnav (https://lnav.org) is phenomenal. Embedded SQLite... easily scriptable... OOB log formats galore, or define your own... it's a mini ETL powertool that scales to at least a few million rows and runs in your terminal. Maintainer's a friendly dude, too.

[clumsysmurf]

Interesting! Do you know how it compares to multitail? (https://www.vanheusden.com/multitail/) They look very similar.

[tstack]

I don't think multitail really understands logs like lnav does, it's just following the last lines in the file. For example, if you try to follow multiple files in multitail like so:

    $ multitail /var/log/install.log -I /var/log/system.log

You get a view with the tail from one file followed by the tail from the other, they are not collated by timestamp. In contrast, if you do the same thing in lnav:

    $ lnav /var/log/install.log /var/log/system.log

You will get a single view with all of the log messages from both files and they're sorted by their timestamps. Here is all of what lnav is doing:

  * Monitoring files/directories for updates
  * Decompressed files/archives
  * Detected the log format for each file
  * Created SQLite vtables that provide access to log messages
  * Built an index of all the log messages in all the files so
    you can jump to a certain point in time or to the
    next/previous error message.
  * Display all log messages with syntax highlighting

[aargh_aargh]

I migrated from multitail to lnav. Turned out to be a no-brainer.

I second the above, just one pain point with multitail to add. I often page/search/filter in the scrollback buffer (I typoed "bugger" - Freudian slip?) and in multitail the scrollback is a separate window with a frame and everything, which is a pain (copying whole lines using mouse includes the frame, ugh). The filtering/searching being a separate pain.

One thing I used in multitail and not sure if I migrated wholly to lnav was log file syntax highlighting using regexes.

[hermitcrab]

Our Easy Data Transform software is intended for data wrangling and desktop ETL. But it has lots of features useful for browsing and manipulating log files, including:

* powerful filtering (including with regex)

* smooth scrolling of millions of rows of data

* support for csv, text, xml, json, Excel and other formats

* available for Window and Mac

[joshka]

Came here to say the same - massive props to the lnav devs.