[ohthehugemanate]

At a minimum you need backup, regular restore tests, and alerts when backups stop or restore tests fail.

Personally I automate restore testing with cron. I have a script that picks two random files from the filesystem: an old one (which should be in long term storage) and a new one (should be in the most recent backup run, more or less), and tries restoring them both and comparing md5sums to the live file. I like this for two reasons: 1. it's easy to alert when a cronjob fails, and 2. I always have a handy working snippet for restoring from backups when I inevitably forget how to use the tooling.

IMO alerting is the trickiest part of the whole setup. I've never really gotten that down on my own.

[Helmut10001]

I recently set up email alerting through the syslog agent from Telegraf-Influx-Grafana, where Grafana is used for Email alerting and InfluxDB for filtering for the specific syslogs.

On another VM, I used postfix to email logs after cronjob (failed or passed), which also works great.

[amjd]

I use ntfy.sh for sending push notifications from scripts and such. It's open source and free (they have paid plans as well now, but I didn't encounter any limitations in the free plan).

Not an endorsement, just a happy user.

[pohuing]

+1 for ntfy: it's also trivial to self host