Show HN: Tailspin – A Log File Highlighter

[bensadeh]

There are multiple programs for working with log files, both inside and outside of the terminal. These tools are feature rich and functional and offer lots of control to the user.

However, a lot of the time I found myself just opening raw log files in `less`. I enjoy the simplicity of quickly skimming through the logs, but it was difficult when the wall of text was in one color.

For this reason, I created a command line tool called `tailspin` (the name is a take on the `tail` command) that did one thing: highlighting of log files. It is configurable, but requires no setup. It defaults to opening up a file in `less`, but can also work with stdin and stdout.

[mh-cx]

Even if it's obvious: I was missing a "How to use" section in the README that demonstrates how this tool can be invoked.

EDIT: Ok, there's something in the "Working with STDIN and STDOUT" section. But TBH I'd find it more helpful to have the different use cases collected at the top. To me it wasn't even clear how the command is called.

[cyberge99]

I would love to see better performance with triggers. Even on an M1 Pro Max when I have triggers I get a slightly noticeable lag. It could just be me though. I run at 5k resolution with a lot of text.

[radlad]

Are triggers a tailspin feature, or are you referring to iTerm triggers which are mentioned elsewhere in this thread?

[bensadeh]

Thanks for the feedback! Good idea to showcase the use cases earlier and clearer.

[joshka]

This is really pretty - I do really wish for a good rust replacement for lnav[1] someday.

1: https://lnav.org/

[tstack]

I also wish for a rust replacement of lnav! (I'm the author)

I actually started working on one earlier this year and made some progress before getting distracted. I was using tui-rs, which then got forked to ratatui. I should check back in with it sometime soon.

[joshka]

As it happens, I'm one of the maintainers of Ratatui, so it's been a little bit more than an idle thought... If you do happen to find time to revive the project I'd be happy to help. Come say hi on the ratatui discord / matrix.

[hiAndrewQuinn]

Did you run into any huge roadblocks when you were trying to port the C++ codebase? I've really been digging `lnav` for the last few days - it gives me a sizeable fraction of Datadog's power in my tiny new embedded world.

[tstack]

Yes, there were some roadblocks and that made me lose some steam. What's kinda funny is that I spent a lot of time trying to make pretty errors[1] in lnav as inspired by rustc, but had a hard time replicating that in the rust ecosystem. For example, I was using serde to parse lnav's JSON configuration files, but getting serde to report line numbers (and/or a JSON path) for errors was pretty difficult. I think serde was also missing a way to report multiple errors.

Otherwise, I had wanted to do some rearchitecting to do log parsing in background threads since it's largely sync in the C++ version. Doing that in rust was great since the borrow-checker guided me down a sane path.

[1] - https://lnav.org/2022/08/04/pretty-errors.html

[joshka]

I've been meaning to have a play with miette for a while:

    fn main() -> Result<()> {
        let config: Config = serde_json::from_str(DATA).map_err(|e| BadJson::from_json_err(e, DATA))?;
        Ok(())
    }

    #[derive(Debug, Error, Diagnostic)]
    #[error("Received some unexpected JSON.")]
    #[diagnostic(
        code(some::app::bad_json),
        url("https://my-app.com/some-json-error"),
        help("Fix the JSON.",)
    )]
    struct BadJson {
        source: serde_json::Error,
        #[source_code]
        json: NamedSource,
        #[label("here")]
        err_loc: (usize, usize),
    }

    impl BadJson {
        fn from_json_err(err: serde_json::error::Error, json: &str) -> Self {
            let offset = SourceOffset::from_location(json, err.line(), err.column());
            Self {
                source: err,
                json: NamedSource::new("config.json", json.to_string()),
                err_loc: (offset.offset(), 0),
            }
        }
    }

Generates:

    Error: some::app::bad_json (https://my-app.com/some-json-error)

    × Received some unexpected JSON.
    ╰─▶ invalid type: integer `123`, expected a string at line 13 column 51
        ╭─[config.json:12:1]
    12 │                             "color": "#f6f6f6",
    13 │                             "background-color": 123
        ·                                                   ▲
        ·                                                   ╰── here
    14 │                         }
        ╰────
    help: Fix the JSON.

[swah]

Another useful trick is iTerm's Triggers - I use those a bit to find specific strings and store in Airtable or highlight in yellow.

[joshka]

Yeah - highlighting ERROR as white on red everywhere anywhere is ops gold.

[Groxx]

You can set up links too! Handy for noticing Jira task IDs and linking them, or (if you work on a single project) GitHub pull request numbers.

[kunley]

"it requires no configuration or setup", still every single example comes with a config file. WTF?

If you look at it more closely, it can be seen that there is no configuration needed for pattern discovery, but there is for colors. Still, it all together looks a bit dumb. I mean, it's pretty un-marketing to claim something and in the following section do the opposite, even if for a different reason.

I believe this unfortunate working can be fixed.

[radlad]

This looks really cool! I interact with log files on the command line often. I usually end up using less unless I need better search / highlighting, at which point I bust open vim.

This looks like a great alternative. There was no Homebrew package so I installed Cargo in order to compile it.

Unfortunately, the first log file I tested it on (admittedly quite large at 868M) took 2 minutes and 45 seconds to open in tailspin, pegging a core the entire time. This same file opens instantly with less.

[RandomBK]

Probably not as efficient, but I've been happily using grc [0] for the past several years. It handles simple rules quite well - beyond the basic info/debug/error coloring I use it for QOL such as different colors for even/odd timestamps and highlighting decimal places in large numbers.

[0] https://github.com/garabik/grc

[miller_joe]

Looks great. Another tool in this space is https://github.com/humanlogio/humanlog

[drcongo]

`apt install tailspin` gives me `Unable to locate package tailspin` - Ubuntu 22.04

[michaelmcdonald]

Same for me - Debian 10

[arp242]

First release is from July this year, so not surprising older Ubuntu and (even older) Debian doesn't have a package.

Anyhow: https://repology.org/project/tailspin/versions

[drcongo]

Thanks

[bensadeh]

Sorry about that! The debian packages are a bit outside my control, but `cargo install tailspin` should do the trick. I might need to remove it from the readme if people aren't able to install that way.

[syoc]

Cool project. Very happy about you taking the time to package it. I never use cli tools I cant install with apt because i log into a lot of servers. Hope i still remember this tool once i leaves sid and gets to stable.

[michaelmcdonald]

Would you be able to add instructions for a source install? That would help alleviate issues with the package not showing up in various repos.

[bensadeh]

Updated the readme now.

[textcal]

Looks pretty for troubleshooting long log files. Thanks for open source it.

[tibanne]

Is it possible to use this in vscode as a plugin?

[thinphp]

does this have security issue when hackers injects malicious code into color codes like they mentioned in the Defcon talk?

[tamimio]

Love it! Thanks for sharing it!

[philo23]

A little bit off topic, but I’ve always wanted a “simple” CLI tool that works exactly like grep (same flags and regex support), but rather than filtering the result, it just highlights the matches in the output.

I’ve not found anything like that yet though, closest I’ve come is using grep with the before/after/context flag with a high number of lines or using search in less, but I want to do something like...

    tail -f access.log | highlight ".jpg"

Maybe I’ve just not found the right flag for grep yet...

Edit: Thank you everyone for your suggestions, you've all been very helpful! And apologies again for hijacking the topic, tailspin looks very neat.

[loudmax]

This is how I colorize with grep:

  cat <file> | grep -E --color "<regex>|$"

Where `<regex>` is your pattern.

[ralgozino]

adding "or an end of the line" to the regex is pretty clever, I'm stealing it :) thanks!

[quickthrower2]

Obligatory “you don’t need cat”

[svedin]

ripgrep has a --passthru flag that does exactly this

  tail -f access.log | rg --passthru '\.jpg'

[philo23]

Brilliant, this looks like exactly what I want. I already use ripgrep a lot, so I should have guessed it had this feature already but wouldn't have known what to search for to find it! Thank you

[zem]

I have that aliased to `hl` in my bashrc

[jpitz]

As does ack

[jstanley]

I wrote this years ago: https://github.com/jes/emph

[pdimitar]

A Markdown README and some screenshots will really help.

[CubsFan1060]

Try

| less -p ".jpg"