[616c]

On a slightly unrelated note, as a largely amateur Linux user: have people made systems that instead of grepping for info, use machine learning do detect normal patterns of a log file (like what type of events, similar, at different intervals) and report the anomalous output via email or report to an admin?

I was thinking this would be a cool area of research for me to try programming again, but it seems so daunting I am not sure where to start.

[nosir33]

I don't know of any systems that do this.

As an software developer, I generally use log levels to indicate severity in my logs. So grepping for ERROR should catch anything I had the foresight to log at the ERROR level.

Simple heuristics like the number of WARN level logs a minute may be useful.

Beyond that it sounds interesting. It may be hard to do in a general way, so focusing on Apache logs or something common may be a simpler task.

[falcolas]

In addition to logging, you can send out a statsd[0] message, graph it, and use something like Skyline[1] for alerting based on trend issues. You can also use logstash to generate metrics on logs when sending them up to Elasticsearch.

[0] https://github.com/etsy/statsd [1] https://github.com/etsy/skyline

[616c]

Excellent sample projects, especially Skyline. This seems the closest thing to what I had envisioned sofar.

Very cool stuff. Do you use it?

[falcolas]

Skyline was a bit too much overhead, but we took the concept and adapted it to our needs.

When I say too much overhead, I'm referring to the carbon proxy and redis requirements. We found that just using the json output from graphite was sufficient to feed a trend monitoring system.

The output is pretty sensitive, moreso than Icinga2 (Nagios) expects, so we had to turn down a few of the "is this really down" re-checks, since it would silence legitimate trend alerts.

[ars]

I use the logwatch program for that. There is no machine learning, it's entirely manual with a large list of things it filters out, but the defaults are quite good.

It emails me any log entires it doesn't know about. I did have to add a large number of ssh lines that it should not bother me about, but other than that it works very well and I find it very useful.

[616c]

Cool tip. I remember hearing the name but knowing it had these features. I will definitely check it out.

[malka]

you can use fail2ban for this. It is used to automatically ban IP that, for instance, tries to bruteforce your SSH, but it really is an engine that match regexp log file lines, and fires an action if the regexp match.

So you can use it for other usages (such as sending an admin a mail if suddenly your server sends 500 errors, or a unusual amount of 404 errors for instance)

[616c]

Of course. Not to be dismissive, but I am familiar with fail2ban. I was wondering if anyone had this idea that did not require manual or pre-set rules, like that the program would go passive for a few days, reading log files and learning certain log entries will be indentical minus timestamp, then some change with a small amount of text in addition, and others have never been seen (or will not match in the next stage). Next stage turns active, and the machine filters down and sends you anything it has not seen over time and knows must be something anomolous.

I like fail2ban, a lot, and alternatives in that field, but when I looked at the Arch Linux package last time there were dozens of commented-out, but heavily commented nonetheless regexp template files like you describe. I think this would be a neat machine learning thing.

What I am going for: use AI to train a passive entry-level sysadmin to warn you.

[madhouse]

I experimented with that, and heard others toying with the idea too. There are even products out there that do something similar.