[holowoodman]

* Services that should be running (enabled/autostart) but aren't. This is easier and more comprehensive than stuff like "monitor httpd on webservers", because all necessary services should be on autostart anyways, and all stuff that autostarts should work or be disabled.

* In our setup, container status is included in this thanks to quadlets. However, if using e.g. docker, separate container monitoring is necessary, but complex.

* apt/yum/fwupd/... pending updates

* mailqueue length, root's mailbox size: this is an indicator for stuff going wrong silently

* pending reboot after kernel update

* certain kinds of log entries (block device read error, OOMkills, core dumps).

* network checksum errors, dropped packets, martians

* presence or non-presence of USB devices: desktops should have keyboard and mouse. servers usually shouldn't. usb storage is sometimes forbidden.