[Bender]

But the real reason I stopped running my own mail is that I didn't trust myself with regular backups (even though I still do them via IMAP on occasion) and disaster recovery. I don't think anyone self-hosting really has this figured out.

On the server itself a cronjob calling rsnapshot [1] is a good habit in the event something gets corrupted or someone deletes mail by mistake and you want to save the day. RSnapshot uses hardlinks to avoid copying the same data to keep the size small which allows for many snapshots. Then a cronjob to call a script that uses either lftp+sftp+mirror or rsync to back up all the important things to a standby node along with a corresponding script on the standby node to quickly copy everything into place and start everything in an idempotent manor can get one back into operation quickly. All of this should be tested quarterly in the event some software update breaks any assumptions.

If running this on VM's in a VPS provider there may be an option in the VPS control panel to swap IP addresses on the active and standby assuming they are in the same region.

[1] - https://rsnapshot.org/rsnapshot/docs/docbook/rest.html

[commandersaki]

Not that helpful if say your server has been attacked by ransomware. Happened to my friend that I was colo-ing with who was running a mailserver at the time. He didn't have proper backups and/or DR (though I think his thick mail client had a copy of most of his mail); he didn't pay and instead moved everything to Fastmail.

[Bender]

If the server has been attacked somehow by ransomware then the sftp backups will be fine. sftp to a chroot sftp-only configuration and rsnapshot running on the remote end means one would have to not notice this for a very long time before all backups are corrupted. I am happy to demonstrate this if need be.

Adding to this time between backups can be shortened by using a different cronjob to utilize inotifywait in a loop and back up to a different or same sftp account achieving both scheduled and ad-hoc snapshots.

[commandersaki]

Yeah then you need a monitoring solution for your backup, another colo for your DR, etc. You end up with all this overhead that you need to always be on top of, unless you can hire someone that you trust and maintain things for you. It's just better value hosting elsewhere where all these unhappy path scenarios have been careful considered and taken care of. I'm definitely not say use free services either as they do not even come with support, but there's middle ground.

[Bender]

In terms of monitoring I could envision this just being a section of the backup script on the primary servers that perform a dry-run backup and if the delta is massive then something has likely tampered with the files, refuses to do a real backup and sends an alert, text message or otherwise. It would have to be something that people would not ignore.

The sftp backup servers in their script that kicks off their rsnapshot could also count total vs new files and alert if nothing has changed or too much has changed. Each person/org would have to determine what is an unusual time to go without changes assuming the primary mail servers have died due to malware or the new file delta is too big due to files all being tampered with.