Restoring from duplicity takes a proportional amount of time related to the backup chain length, as archives are stored as forward-deltas.
Nothing against forward deltas, but just consider that:
- restoring time increases with increment count
- risk of corruption of a single archives will make all further increments worthless.
These are very important things to consider when doing a backup.
Currently, you have to limit the chain length by performing a full backup every N increments (with N being as short as possible), which defeats the purpose of efficient increments.
I have requested the ability to specify manually the ancestor of the increments (by time or count), so that one could implement a non-linear hierarchy with multiple levels like one normally does with tar/timestamps, but the request was dubbed as "unnecessary given the delta efficiency" (despite the fact that efficiency is just one variable). Having a 3 level backup (daily, weekly, monthly) would make duplicity much more space efficient, reduce the number of full backups needed, and shorten the chains to the level that would make restores of "last year" actually _possible_.
I sent several patches to fix duplicity behavior with files larger than 1gb (by limiting block counts), which got integrated, but are still a far cry to make duplicity work decently as a whole-system backup&restore solution. It's just too slow. And like you said, several bugs afflicted duplicity in the past that would make restore fail in many circumstances. I also debugged my share of issues, which led me to think that very few people actually tried to restore from arbitrary increments with duplicity and/or used them to archive a large system.
Many got fixed, but I won't consider duplicity again until I can control the incremental ancestor and reduce the chain lengths (and it's silly to think that "rdiffdir", distributed with duplicity, would allow for that easily).
Nowdays I use rdiff-backup, and use a second script to tar/encrypt the deltas after the backup has completed.
I'm keeping an eye on "bup" (https://github.com/bup/bup), but I cannot backup "forever", thus without the ability of purging old versions it's only useful in a limited set of cases.
> I'm keeping an eye on "bup" (https://github.com/bup/bup), but I cannot backup "forever", thus without the ability of purging old versions it's only useful in a limited set of cases.
I wrote ddar before I knew about bup. It doesn't have this limitation; you can arbitrarily remove any archive. However, it does not do encryption, so I wouldn't recommend using it to store on S3 directly.
I actually tried again today, and this is still promiment in the limitations:
bup currently has no features that prune away old backups.
Because of the way the packfile system works, backups become "entangled" in weird ways and it's not actually possible to delete one pack (corresponding approximately to one backup) without risking screwing up other backups.
Thanks for pointing that out. Encryption is not really an issue for me. It's very easy to perform encryption using a encrypting FUSE filesystem or afterwards when needed.
Nothing against forward deltas, but just consider that:
- restoring time increases with increment count - risk of corruption of a single archives will make all further increments worthless.
These are very important things to consider when doing a backup.
Currently, you have to limit the chain length by performing a full backup every N increments (with N being as short as possible), which defeats the purpose of efficient increments.
I have requested the ability to specify manually the ancestor of the increments (by time or count), so that one could implement a non-linear hierarchy with multiple levels like one normally does with tar/timestamps, but the request was dubbed as "unnecessary given the delta efficiency" (despite the fact that efficiency is just one variable). Having a 3 level backup (daily, weekly, monthly) would make duplicity much more space efficient, reduce the number of full backups needed, and shorten the chains to the level that would make restores of "last year" actually _possible_.
I sent several patches to fix duplicity behavior with files larger than 1gb (by limiting block counts), which got integrated, but are still a far cry to make duplicity work decently as a whole-system backup&restore solution. It's just too slow. And like you said, several bugs afflicted duplicity in the past that would make restore fail in many circumstances. I also debugged my share of issues, which led me to think that very few people actually tried to restore from arbitrary increments with duplicity and/or used them to archive a large system.
Many got fixed, but I won't consider duplicity again until I can control the incremental ancestor and reduce the chain lengths (and it's silly to think that "rdiffdir", distributed with duplicity, would allow for that easily).
Nowdays I use rdiff-backup, and use a second script to tar/encrypt the deltas after the backup has completed.
I'm keeping an eye on "bup" (https://github.com/bup/bup), but I cannot backup "forever", thus without the ability of purging old versions it's only useful in a limited set of cases.