Hacker News new | ask | show | jobs
by voltagex_ 3926 days ago
If I was going to tar and then gpg files, what would be the optimal "chunk size" from your point of view? Say I have 10GB that I'm encrypting and then splitting up, are you going to want 1GB chunks or many more 100mb chunks? (This also raises the question of how much data I want to lose to corruption...)
2 comments

I would recommend 1 GByte chunks for several reasons. The first reason is that you'll get very good throughput and efficiency at that large size. If you go as small as 10 MBytes we see long distances to our datacenter not "ramping up" fully. In other words, we can't get a single threaded application between New Zealand and California to get the advertised bandwidth for small chunks, but up at 1 GByte we DO get the advertised bandwidth. But if you go too large (like 5 GBytes) we have seen that you start seeing too high a percentage of uploads rejected because of bit errors occurring somewhere in the networking between your computer and our servers (the SHA-1 checksum won't match).
One thing to consider is using ecryptfs as I do for my data at rest. It's mounted as a (mostly) regular filesystem at ~/Private (or similar) and the encrypted files that serve as the backend (~/.private) are then uploaded to the cloud service of my choice. Currently using crashplan, but B2 would be nice to create archives/snapshots of my data instead of just syncing it as most consumer backup solutions do.

This is a very low friction way to have accessible data and have it encrypted by a fairly popular mechanism.