logcheck, is a really old collection of bash scripts that are surprisingly great for monitoring a *nix server.
It’s great because it’s really lightweight and easy to set up compared to most modern logging and alerting stacks.
It can do this because it works in reverse to how most logging tools work. Instead of trying to find the important stuff and alert on that, it just filters out everything “standard” and alert on everything else.
On a normal, low activity server like my personal one, the standard logs (excluding noisy stuff like web logs) are generally very uniform and boring. And the maintainers and contributors of logcheck have spent quite some time to pre-write filters for all the standard noise which applications put into their logs as part of daily functioning.
I’d recommend everyone who runs their own servers to give it a try. The only annoying part is writing the ignore rules for the stuff that is not yet ignored, but I’m vibe coding a solution for that, for another blog post.
I will now explain how to install on openwrt, which is interesting and useful if you can’t dnf or apt-get it, if you can, do that instead. I’ll use the Turris Omni as example because I have one, but instructions should work for all openwrt and similar. You just need bash, msmtp (or similar) and some cron.
Turris
I have a Turris Omnia, which is a nice router running a variant of openwrt. A long time ago the USB dongle I had in it started throwing lots of errors and I was none the wiser until I happened to login to it by coincidence and saw the errors in the log.
Nov 11 09:40:04 turris kernel: [525532.234506] BTRFS error (device sda): bdev /dev/sda errs: wr 709, rd 1629, flush 0, corrupt 0, gen 0
So wanting get some kind of generic alerting, and having good previous experience with logcheck I thought I would try to get it installed. But logcheck is not in the normal openwrt package repository, so I had to go look until I found a blog post by Glen Pitt-Pladdy, who had made a simple logcheck in bash, which works on OpenWRT back in 2011. Here is what I did.
Installing and configuring
Start with downloading mini_logcheck.sh
If you have SMTP, great, if not then you’ll need to create one. I prefer mailgun.com but there are a lot of providers that have free low or restricted tiers.
Modify /etc/msmtp so it looks something like this. Replace the host with your smtp host.
# Example for a system wide configuration file
# A system wide configuration file is optional.
# If it exists, it usually defines a default account.
# This allows msmtp to be used like /usr/sbin/sendmail.
account default
# The SMTP smarthost.
host smtp.eu.mailgun.org
tls on
tls_trust_file /etc/ssl/cert.pem
port 587
from turris@YOUR-DOMAIN
auth on
user <YOUR-SMTP-LOGIN>
password <YOUR-SMTP-PASSWORD>
# Construct envelope-from addresses of the form "user@oursite.example".
#auto_from on
#maildomain oursite.example
# Use TLS.
#tls on
#tls_trust_file /etc/ssl/certs/ca-certificates.crt
# Syslog logging with facility LOG_MAIL instead of the default LOG_USER.
syslog LOG_MAIL
Create
mkdir /etc/logcheck.d.ignore/
edit a file inside that to create a rule, historically, to organise rule files they are split per process/daemon, but unless you have lots of rules I prefer to keep them in one like so
here is a sample to get your started
/etc/logcheck.d.ignore/rules:
odhcpd[[0-9]+]: DHCPV6
cron[[0-9]+]: \(root\) CMD
kresd[[0-9]+\]: $
kresd[[0-9]+\]: > hints
kresd[[0-9]+\]: \[result\] => true$
99-dhcp_host_domain_ng.py
Then edit/create
/etc/cron.d/logcheck with the following
MAILTO=""
36 * * * * root /root/mini_logcheck > /dev/null
Testing rules in logcheck
The hardest thing in my opinion is writing correct rules, the regex grep uses is basic and I generally have to use quite a bit of trial and error to get more complex rules to match.
mini_logcheck does not support the test mode that the normal logcheck does, so I quickly hacked together a test_logcheck script based on mini_logcheck. It can be found here https://gist.github.com/gnyman/a4d7ad7c13113dd9c3fa74442e42c17c
It will test the rules and display any matching lines. So you can modify your rules and re-run the script to see if it matches.
Blaugust
This is another blaugust post. It’s a draft I had lying around but it has not received enough editing or spell checking to graduate from that, so it’s a #draft still.