[galonk]

A lot of people sing BB's praises but I never had a good experience with them. The client was always slow, buggy, and resource hungry, and its UI is terrible. They got shirty with me for reporting bugs when I was using a macOS beta. And finally, at some point even though nothing about my computer changed (it was a Mac Mini, what was going to change), I got a message saying some security/copy protection system had detected that my computer was "different", and I had to un-install and re-install the entire app to fix it (there apparently being no easier way to unset a flag). I uninstalled and skipped the second part.

Instead of using BB, get a Synology/Qnap/FreeNAS box to backup all your stuff locally, and back that up to another service (e.g. Glacier or Synology's own C2).

[willis936]

I caution the casual reader against glacier. It's not what it appears at a glance. Your files should be put into a single archive before upload otherwise you'll spend weeks waiting for AWS scripts to manage old files.

B2/S3 is what most people want.

[linsomniac]

We have 23TB of images stored in S3 and I was recently looking at moving them to Backblaze to save hundreds of dollars per month. These are all individual image files, because reasons.

Then I realized that S3 Glacier and Deep Archive were even less expensive than B2. I took a bit further of a look and found that Glacier/DA files have some fairly chonky metadata that must be stored in normal S3, and for a lot of our images the metadata was larger than the image in question. So Glacier/DA would increase our storage costs. Over all it probably wasn't a money-saving situation.

The ideal use case is to bundle those up into a tar file or something and store those large files, and manage the metadata and indexing/access ourselves.

So, using rclone to copy 11TB of data to B2.

[nijave]

Wasabi is also a little cheaper than AWS S3 afaik

[sliken]

Keep in mind when you create something in wasabi you pay for it for 3 months, even if you delete it minutes later.

[iJohnDoe]

> otherwise you'll spend weeks waiting for AWS scripts to manage old files.

Can you elaborate on this part?

[willis936]

Every file uploaded is considered an immutable archive. It does not have a version history. So let's say you have 100,000 files you backed up and want to update them and don't want to pay for the storage of the old files. You need to request for a manifest of hashes for all files. This will take a few days to generate then you will be given a json file that is over a gigabyte. Next, you will write a script to delete each file one at a time, rate limited to one request per second. Have fun.

[sreitshamer]

Are you maybe referring to Glacier "vaults" (the original Glacier API)? With the old Glacier Vault API you had to initiate an "inventory-retrieval" job with an SNS topic etc. It took days. Painful.

But these days you can store objects in an S3 bucket and specify the storage class as "GLACIER" for "S3 Glacier Flexible Retrieval" (or "GLACIER_IR" for S3 Glacier Immediate Retrieval or "DEEP_ARCHIVE" for S3 Glacier Deep Archive). You can use the regular S3 APIs. We haven't seen any rate limiting on this approach.

The only difference from the "online" storage classes like STANDARD, STANDARD_IA, etc is that downloading an object with GLACIER/GLACIER_IR/DEEP_ARCHIVE storage class requires first making it downloadable by calling the S3 "restore" API on it, and then waiting until it's downloadable (1-5 minutes for GLACIER_IR, 3-5 hours for GLACIER, and 5-12 hours for DEEP_ARCHIVE).

[spacedcowboy]

I just bought a Quantum Superloader3 last Xmas. Each LTO-8 tape (it can take 16 in 2 magazines, I use 15 + 1 cleaning tape) will hold 12 TB without compression, 30TB with, and 7 of them can back up the 100TB used on the 128TB of disk that is the house RAID array.

It takes about 2 days to make a full backup, and I can fit incrementals for the next 5 days on the batch-of-7. Then I switch to the second magazine, and do the same thing. I actually have 3 magazines, one of which I swap in and out every week, and during the before-times, I'd take that off-site to work.

I have ~30 years of data, from back when I was in college and writing CD-ROMs for backup, all on the one system. Admittedly, the major space-taking thing is the Plex library, but I wouldn't want to lose that either. It takes about 5 minutes to walk into the garage (where the server-rack is), swap magazines and I'm done - the rest is automatic.

I have vague ideas for writing a more-efficient tar designed for this specific type of setup (big disk with attached tape). The best way to do it I think is to have multiple threads reading and bzip2-compressing data, piping blobs through to a singleton tape-writer thread. Every now and then (50GB, 500GB, 1TB ?) close the device and reopen the non-rewindable device to get a record-marker on the tape, and then store the tape/record-marker/byte-offset etc. into a SQLite database on the disk. That way I'd get:

- High levels of compression without making the tape head wait for the data, which ruins the tape head. Multiple threads pooling highly-compressed data into a "record"

- fast lookup of what is where, I'm thinking a SQL LIKE query syntax for search, against the disk-based DB. No more waiting for the record to page in from the end of the tape.

- fast find on-tape, since you'd know to just do the equivalent of 'mt fsf N' before you actually have to start reading data

Right now, tar is good enough. One of these days when I get time, I'll write 'bar' (Backup And Restore :)

[sponaugle]

Interesting. Would you mind if I asked you a few questions off-thread about the QS3? I'm working on adding a local tape option to my homelab (https://www.reddit.com/r/HomeDataCenter/comments/pira9v/powe...)

I need a solution for backing up the stored states for my 100 trillion digit PI calculation efforts.

[kingcharles]

> I need a solution for backing up the stored states for my 100 trillion digit PI calculation efforts.

O_O

[spacedcowboy]

Sure, fire away. I didn't know HN even had a PM feature :)

[simplyaccont]

small question about superloader3. how noisy it is ?

[spacedcowboy]

Too loud for an office. Which is a shame because in these pandemic times, I repurposed the shed at the bottom of the garden as an office (insulated, add power and a/c) and I wanted it there.

It's not noisy when writing to tape, but the mechanism is noisy when a tape is being loaded, the magazine is being shuffled to get the right tape, etc. It's the mechanical parts rather than the tape drive itself that's too loud, especially with Webex conferencing being a part of the day now.

So I have it set up in the garage, in the server-rack. I was worried about temperatures in the Summer, so I bought a 100W solar panel, an attic fan, and linked them up, positioning the fan above the rack. That fan shifts so much air that the in-rack fans (with temperature monitoring) didn't get above 85 all summer, which is pretty amazing for the Bay Area. The tape deck seems to be fine in that sort of temperature, and yes I do do the occasional 'tar tvf' to check the data is readable :)

[simplyaccont]

Thanks. Are there any continuously running fans when it's not actively working ? Or something else that makes a noise ?

I got an HP IP KVM switch a while ago. When it's on, it makes as much noise as a bunch of servers or a blade center. Got some fans for retrofit...

[spacedcowboy]

Yes. I'd forgotten about that. There is a reasonably-above-ambient-noise fan that is running constantly. Another reason it was banished to the server-rack in the garage.

[simplyaccont]

one more, did you get it at full price or you found it somewhere at discount/used ? i been trying for a few years, on and off to find something

[k8sToGo]

You need to differentiate between BB Personal Backup and BB B2 service which is more like something you suggested. But these days I just use rsync.net + Wasabi + Kopia + rclone.

[ericcholis]

I really want to use rsync.net, but the price per GB scares me off.

[john-shaffer]

I have one of their borg accounts, which is a fraction of the normal price. https://www.rsync.net/products/borg.html

[rsync]

Please email us about the "HN Readers Discount" which we have offered for 12-ish years ...

[KennyBlanken]

Do you understand that popping up like Beetlejuice every time someone mentions rsync pricing and being snarky about an unpublished discount is not a good look?

Particularly given you never address the commenter's point, which is that your pricing is pretty expensive?

"Price high, low volume" is certainly a valid pricing strategy, but you have no right to be snarky when people say you're priced high.

The fact that people are consistently describing you as overpried means your marketing really isn't showing the corresponding value to them.

[mekster]

Not sure if they're overpriced when they also have borg discount and rsync.net is one of the only service that I can borg or "zfs send" directly as well as the fact that they've been running nearly 20 years? to know they don't disappear next year makes it rather feel not overpriced.

[Osiris]

I have 3.5TB backed up to Backblaze for $5 a month. I haven't found any other online backup option that provides anywhere near that cost.

I use a RAID 1 to handle drive failure and also keep local backups on a NAS. BB is my third layer of backup. I've never run into issues with BB backups so I'm happy for what I get for the price.

[markdown]

> I have 3.5TB backed up to Backblaze

The question this thread seems to be raising is... do you? Do you really? Are you sure?

[janitha]

S3 Glacier Deep Archive, $0.00099 per GB per month.

I have a ZFS based NAS. And periodically do a incremental backup (zfs send) of the entire dataset, encrypt it gpg and pipe it straight up to S3 deep archive. Works like a charm.

The catch with S3 deep archive is if you want to get the data back... It's reliable, but you will pay quite a bit more. So as a last resort backup, it's perfect.

[mekster]

Can you tell how it's done specifically? Are you zfs send'ing to another directory and encrypt it entirely on the fly while transferring to Glacier?

Does it do incremental backup transfer to Glacier or does it have to transfer the entire encrypted blob every time?

[janitha]

No intermediate directory/file at all, all done on the fly.

    sudo zfs send -i <LAST_BACKUPED_SNAPNAME> <CUR_SNAPNAME> | gpg -e -r <YOURKEY> --compress-algo none -v | pv | aws s3 cp --storage-class DEEP_ARCHIVE - s3://<PATH_TO_DEST>.zfs.gpg

The very first time you do it, you will need to do a full backup (ie. without the `-i <...>` option). Afterwards, subsequent backups can be done with the -i, so only the incremental difference will be backed up.

I have a path/naming scheme for the .zfs.gpg files on s3 which include the snapshot from/to names. This allows to determine what the latest backed up snapshot name is (so the next one can be incremental against that). And also use when backing up, since the order or restore matters.

[luckman212]

But how do you verify or test your backups in this scenario?

[janitha]

Pretty much the exact reverse of backing it up https://news.ycombinator.com/item?id=29541729

    aws s3 cp s3://... - | gpg -d ... | zfs recv ...

When restoring the order of restores matters, you first need to restore the full snapshot, and then the subsequent incremental ones in order.

[luckman212]

I meant, how to test them without incurring a large cost.

[janitha]

Ah gotcha, I haven't done full restore of my main dataset.

I've only verified with a smaller test dataset to validate the workflow on s3 deep archive (retrieval is $0.02/GB). I've done full backup/restore with the zfs send/gpg/recv workflow successfully (to a non aws s3 destination), and used s3 for quite a long time for work and personal without issue, so personally I have high confidence in the entire workflow.

[kingcharles]

If you want to try another sketchy service, I use iDrive which was $9.95 for 10TB for the first year.

[mlangenberg]

> get a Synology/Qnap/FreeNAS box to backup all your stuff locally, and back that up to another service (e.g. Glacier or Synology's own C2).

I'm not a big fan of backing-up a back-up and opted for a Time Machine backup to a local nas and in parallel, an off-site backup to B2 with Arq on my macs.

[doublepg23]

I have a Synology NAS currently and use their HyperBackup hosted tier. Maxes out at 1TB sadly, not even enough for my laptop backup. C2 on the other hand is well priced for bulk storage but does not have the “free” deduplication.

[hedora]

I use HyperBackup to backup multiple TB to Backblaze B2. Works great, FWIW.

[doublepg23]

The hosted version I'm thinking of is their Synology C2 personal it seems.

Currently I use it to back up my lossless music collection and nothing else.

Backblaze B2 was on the table too, however I think my asymetric internet connection is my biggest issue right now. Only 40-50Mbps upload won't do much for backing up multiple TBs of data. May need to consider pre-seeding drive option if I can justify the cost.

Other solution was a separate, lower spec Synology I can pre-seed and send to a friend's house who has a homelab.