[exhilaration]

> automated test restore failed

You automatically test restore? That makes sense but I've never heard of that before, can you describe the process?

[semi-extrinsic]

Not OP, but I would guess it's something like this:

  1. Make a e.g. 30MB file of random data  
  2. Copy it to "_reference" file  
  3. Upload the file to backup service  
  4. Restore the file from backup service  
  5. Diff restored file against reference

[chanandler_bong]

Pretty simple, really.

Pick a couple random files that should be in the repo, restore them from a random archive, check the md5sums against the source. If the md5sums don't match (or the file can't be found), something is wrong. I am mainly backing up RAW image files, so they should never change.

Basically...

$TEST_FILE=$(ls -p /source_dir | grep -v / | shuf -n1)

$TEST_ARCHIVE=$(borgmatic -c config.file list | shuf -n1)

borgmatic extract yada yada yada

md5sum $TEST_FILE restored_file

[adobrawy]

I don't use borg, but I used duplicity, which offers something like that. The verify operation simulates a backup restore to compare whether the restored file's checksum matches that expected from the metadata and optionally against the local file. I use this routinely, interesting to see that a local S3 provider can sometimes mess up your files silently.