[rdtsc]

Yap borg is the first thing I thought of. It already does a lot of this an more: encryption, configurable encoding, a rolling hash computed by the Buzhash algorithm and so on.

Maybe it wasn't geared for CDN delivery during restores but otherwise I've been impressed by borg so far (haven't deployed it in production, only played with it locally though).

https://github.com/borgbackup/borg

This is a description of the internal design:

http://borgbackup.readthedocs.io/en/stable/internals.html

[dom0]

The "latest" version of that page has seen significant additions: http://borgbackup.readthedocs.io/en/latest/internals.html

[rdtsc]

Thanks. That is a better reference indeed. It's got nice diagrams as well. Can't edit my post any longer, so hopefully others will just see your message.