[malka]

you can use fail2ban for this. It is used to automatically ban IP that, for instance, tries to bruteforce your SSH, but it really is an engine that match regexp log file lines, and fires an action if the regexp match.

So you can use it for other usages (such as sending an admin a mail if suddenly your server sends 500 errors, or a unusual amount of 404 errors for instance)

[616c]

Of course. Not to be dismissive, but I am familiar with fail2ban. I was wondering if anyone had this idea that did not require manual or pre-set rules, like that the program would go passive for a few days, reading log files and learning certain log entries will be indentical minus timestamp, then some change with a small amount of text in addition, and others have never been seen (or will not match in the next stage). Next stage turns active, and the machine filters down and sends you anything it has not seen over time and knows must be something anomolous.

I like fail2ban, a lot, and alternatives in that field, but when I looked at the Arch Linux package last time there were dozens of commented-out, but heavily commented nonetheless regexp template files like you describe. I think this would be a neat machine learning thing.

What I am going for: use AI to train a passive entry-level sysadmin to warn you.