[leejoramo]

Arq has been part of my backup systems since version 2. This upgrade looks very good. Especially the expansion beyond relying solely on Amazon S3 and use number of other services including SSH.

Having used the past Glacier support, the new S3 Glacier Lifecycle will be much better.

I am wondering about when (if?) the open source arq_restore and format documentation will be updated.

[OWaz]

Do you mind elaborating what your total backup solution is? I've been looking at Arq but am not aware of what it does not do and what I would need to fill in the gaps. Currently I rely on time machine backing up to three different drives and keeping one of those drives in the office.

[leejoramo]

I will ignore what I do for server backups, and I will focus on my MacBook. I run my web development business and my personal life off of this system.

The software I use: CrashPlan, Arq, DropBox, Carbon Copy Cloner.

CrashPlan: I have backing up my data, configuration /Users, /Library. I have a bunch of regex restrictions to not backup various files such as caches, VMWare images, etc. Crashplan backups up to their cloud servers every 15 minutes. When I am at home (where I work from) it also backups to a local copy of CrashPlan on a server.

Arq: I am doing daily backups to Arq. These are now being done to Glacier for long term / last resort backups. This only backups my /Users with heavy restrictions on which files.

DropBox: I have many of my documents stored in Dropbox with the PackRat feature to keep copies of every version and deletion. I don't consider DropBox to be backup by itself, but I often find it is much faster to find and restore something via Dropbox than other methods. I also take care about the types of data I put on Dropbox.

Carbon Copy Cloner: as I mentioned in another part of this thread, I think SuperDuper is better for most people. However, I do use CCC's ability to remotely do a boot able bare metal backup to my home office server. When I travel, I typically take an external backup drive with a current mirror of my system.

I don't use Apple's Time Machine. I think it is a good choice for most home users. As Apple has added more features to Time Machine, I do think about adding it to my mix.

That covers most things. I do have somethings under SVN or Git which could be considered another layer of backup.

Currently, the biggest pain point for me in backups is VMWare images. I currently have 4 Linux and 3 Windows images on this system, and they can cause a huge amount of data needing to be backed up every time they are used.

[Stratoscope]

> Currently, the biggest pain point for me in backups is VMWare images. I currently have 4 Linux and 3 Windows images on this system, and they can cause a huge amount of data needing to be backed up every time they are used.

This is where a sector-by-sector backup program shines.

I don't know what to recommend on the Mac, but on Windows, ShadowProtect is pretty wonderful. It backs up only changed sectors - update 10MB in a 10GB file and it only copies that 10MB - and it's insanely fast.

Even with a file-by-file backup, one thing you can do for VMware images is to take a snapshot. After you take a snapshot, further changes to the VM don't go into the large .vmdk file you just snapshotted, they go into a new, potentially much smaller .vmdk file, so your next incremental backups may be much smaller.

[ojilles]

Why not just setup backups from inside the VM's, while having the base VM image backed up somewhere (once) as well?

[leejoramo]

I have sometimes done this with Linux VM's that I use for local development.

Usually, the VM disk images are some where between the need to backup Applications and configuration files, and not as important my work and data files.

The problem with VM's is not just the quantity of data that needs to be backed up, but the overall size of the data that needs to be evaluated. I think that CrashPlan does a pretty good job of just coping the changed data of the disk image, but it has to do a HUGE amount of processing with every backup. Therefore VM's are hard to fit in with the remote versioned backups of CrashPlan and Arq.

I do backup these up via Carbon Copy Clone when I mirror the entire drive.

[Stratoscope]

> Why not just setup backups from inside the VM's, while having the base VM image backed up somewhere (once) as well?

That would be quite a project, compared with backing up everything on the host system as I do now.

I have all sorts of VMs. Some of them are extremely minimal OSes (think router/firewall distros). I have no idea how I would be able to back these up from inside the VM. And even for the VMs where I could do that, why bother? It seems like a lot of work.

By having an extremely fast sector backup running on my host system, I can be sure that all of my VMs are backed up, with no extra effort when I install a new one. I don't have to worry about how I would do a "restore" in any of those specific VMs, I can just restore files on the host OS and know that it will work perfectly.

A much simpler and more convenient solution.

[chmars]

@leejoramo:

> DropBox: I have many of my documents stored in Dropbox with the PackRat feature to keep copies of every version and deletion.

Aren't you worried about the lack of encryption?

[leejoramo]

Due to privacy concerns, I don't put everything in Dropbox. But most of what I do doesn't need to be that protected. And I do keep somethings in Dropbox that have been gpg encrypted.

I live in a office and work environment with windows that people can look in. I don't much like people staring in, but I am also not about to keep the blackout curtains drawn at all times.

I do wish that there were other more secure options than Dropbox, but the combination of easy of use, price, third party support, and collaboration make Dropbox hard to beat.

[cmsj]

So, you need to cover yourself against (at least) the following:

1. Bug in your backup software This is addressed by using more than 1 piece of backup software.

2. Corruption in your live data (i.e. your filesystem corrupts your favourite baby photo) This is addressed by having lots of incremental backups going back into history. Note that Time Machine throws away historical incrementals over time, so does not protect against this, given long enough time windows.

3. Failure of your backup hardware This is addressed by using more than 1 piece of backup hardware.

4. Destruction of your backup hardware This is addressed by having your backups exist in more than 1 physical location, so you can never lose your live data and all your backups because of, say, a house fire.

5. User error deletion of data This is addressed by having backups that run frequently.

My strategy is:

* Time Machine to a Time Capsule on my LAN

* Time Machine to an external disk on my Mac

* Nightly Carbon Copy Cloner clone of my entire disk to (the same) external disk on my Mac

* Nightly Arq backup to Glacier's Ireland location (I live in London)

So (in addition to the live copy of my data on my Mac's main disk) I have 4 copies of my data, from 3 different pieces of backup software, on 3 different pieces of hardware, in 2 different locations. The CCC clone is there mainly because it's bootable, so if my mac's SSD fails, I can reboot and hold a key and I'm no more than 24 hours behind.