[pazra]

Backblaze is such a great company. Not only do they offer almost unlimited backup for just $5 a month, they also publish extremely useful articles such as this one.

Good to see Seagate has improved the quality of its product significantly, but a 3.5% failure rate still seems rather high. How old are the drives in question?

[theandrewbailey]

Does anyone think that these reports have shamed manufacturers into decreasing failure rates?

When the FCC started doing reports on internet speeds, ISPs took notice and in the second report, an ISP's actual speeds better matched advertised speeds. It would be awesome if these Backblaze reports were likewise improving hard drives.

https://www.fcc.gov/reports-research/reports/measuring-broad...

[click170]

I'm actually wondering if one of the reasons that they're having trouble buying WDC drives is because WDC keeps trying to add a clause into the purchase agreement saying BackBlaze can't release information about WDC drive failure rates. Anyone from BackBlaze able to comment on that?

[brianwski]

Brian from Backblaze here - so far none of the drive manufacturers have done anything like that. Part of it is the manufactures flatly refuse to let us purchase from them directly (based on we need an order of 10,000 of the same drive in one purchase to buy direct). So the manufacturers have zero control. We always go to one of the hundreds of resellers or distributers out there (like CDW or Ingram Micro). Since we buy from the resellers, the resellers just want to make money, the resellers don't care if we report drive stats.

[jedberg]

Have you guys thought about becoming a reseller as another line of business? In the post it says you need about 1200 drives a month (if read that right) so you could buy 10,000 from the manufacturer, and then sell the extra as Backblaze Approved(tm) or some such. I'd much rather buy a drive I know you guys have put through its paces, especially if I knew that purchase would help fund more data and research, by supporting your business.

[brianwski]

I have suggested it. :-) There is no transparency when purchasing drives, in that we don't know what percentage of the price of the drive goes to the reseller (like CDW) and what percentage goes to the manufacturer (like Seagate). I proposed becoming a reseller just to cut CDW out and pocket the profit margin that CDW lives on.

DISCLAIMER: I have not run my brilliant idea past any lawyers, so it may not be legal. We have not done this (yet).

[Terr_]

"Consider: Morningstar rates funds. Imagine: Backblaze rates drives." :P

[wyldfire]

> buy a drive I know you guys have put through its paces

Are you talking about some kind of burn-in at Backblaze for units sold to get past the infantile failure region? Or just Backblaze-as-drive-retailer but they specifically endorse this model because they have good experience with it?

[qaq]

WDC is likely not selling directly at such low qtys and if they do pricing would not be good.

[tw04]

Good to see Frontier wasn't phased and actually went down. I'd imagine if you checked 2016 they'd be closer to 60%.

[saryant]

How's the restore process?

I've actually been backing up to Backblaze for about a year now but (knock on wood) haven't had to restore any data. That said, one of my drives has been acting up in the last few days so that moment may be at hand (though I also have it backed up to a second local drive).

[ansible]

In addition to what Yev has said, I'd say:

You don't have backups, unless you can do restores. So you have to practice doing restores.

For my servers at work, we're using btrfs snapshots and send/receive to the backup host. So restoring files is just going into the appropriate snapshot directory, and copying out the files of interest.

If your backup scheme is any more complicated than that, you need to practice it at least a few times per year so that it is completely familiar.

Hilarious story from the old days...

We were doing backups to QIC tape drives. At one point, there was a lightning storm. The servers were plugged into UPSs with power protection though.

However, when running a backup, I noticed that the tape drive sounded a little different. So I check one of the backup tapes... the tape drive would no longer switch over the tracks on the tape. So it was just overwriting the same track again and again. Corrupted backups. Worse yet: silently corrupted backups. No messages from the OS about a hardware problem.

That could have been bad news if it wasn't caught quickly.

[atYevP]

> You don't have backups, unless you can do restores. So you have to practice doing restores.

100% - I worked hard to make sure that was in the Best Practices sent out to every person that signs up with Backblaze. Restoring is the most important part. So far we have over 200PB of backups, but the stat that I like even more is that we have restored over 10 Billion files.

[existencebox]

I realize this is slightly off topic, but I want to nerd out for a moment re: your comment on hearing something wrong with the tape drive; a skill I always felt was under-recognized for how much of a "superpower" it gives you, that being how critical sound is for a good sysop. Broken AC belts, bad hard drive backplanes, boot cycles, all things I've run into where the sound was the cue; detecting an unalerted tape drive failure is the icing on the diagnostic case.

[LeifCarrotson]

Audio is an incredibly rich feedback mechanism for all kinds of mechanical processes. And the fascinating part about it is that our brains process it so effortlessly. If the data your ears can analyze from your car were presented visually, perhaps as a scrolling FFT spectral graph or plots of a host of sensors, you'd never notice a momentary misfire or a tiny change in pitch. It would be complete data overload! But even untrained ears can pick out errant noises.

[ansible]

I had another incident like that earlier in my IT career.

I was a 'terminal room consultant' in college... back when we had serial terminals hooked to Unix systems. Part of the job was the care and feeding of a couple printers, a big ol' line printer (green bar paper) and a Printronix graphics printer (dot matrix, for printing out fancy lab reports you wrote up using troff).

So over time, from loading paper and clearing jams, I had accumulated hours and hours of hearing these two guys chatter as they went about their business.

At one point, I noticed that the Printronix printer sounded funny. Just off, in some way. So I call it in for maintenance, but they don't seem to care what an undergraduate punk thought about printer sounds.

Sure enough, a week later, I see it is down and taken apart for repairs.

Your ears, your nose, all your senses should be used for debugging and general investigation.

[linker3000]

Here in the HGST EMEA lab, you will often find an engineer listening to a drive spin up with an induction pick-up and amplifier, muttering something like "Yep, this one's running firmware XYZ", or "Hmm, sounds like this one has the older, unmodified ramp".

[matwood]

> You don't have backups, unless you can do restores.

Bingo, I was just explaining this to someone yesterday. Testing the restores MUST be part of the backup strategy. If your db data is small enough to have it all in your test environment, I often try to test the restores by restoring to the test db and then using that db for the test environment until the next test restore.

[drumdance]

I had a drive fail a couple years ago and restored it with Backblaze. The one gotcha is it wasn't a bootable backup, so it was still a pain getting my system back to something approximating what it was before. Since then I've added a weekly bootable backup to a local USB drive. Not failsafe but good enough for my needs.

[atYevP]

> I've actually been backing up to Backblaze for about a year now but (knock on wood) haven't had to restore any data.

Yev from Backblaze. Please test a restore. We have that as part of our best practices. Why? Hopefully you won't but if you DO need a restore, it's likely that you'll be in a heightened state of panic, so familiarizing yourself with the process before hand make it go more smoothly! It's pretty easy, though we are currently working on ways to streamline it even further :)

[brianwski]

Brian from Backblaze here. Let me elaborate on what Yev said.

When customers are doing a restore, almost by definition they are not having a good day. For example, this could be somebody who just had a $1,200 laptop stolen, and now they might lose every photo they have of their child who died last year. Real example. :-( So they show up to the Backblaze website freaked out of their minds, and they FORGET THEIR PASSWORD or something silly and minor like that, and after guessing a few times our support guys get a flaming hot chat session with a person using more four letter words than not accusing Backblaze of not having their restore.

When we resolve it all (help them with the "recover password" feature) then we usually get a happy customer for life who apologizes for losing their temper earlier. I always find it amusing when they think they are the FIRST customer to ever lose their temper under such a stressful situation. Usually it isn't even the first one THAT DAY.

TL;DR - you only restore when you are freaking out. And that's Ok - your worst day is the day when Backblaze has to be the best.

[atYevP]

Yev from Backblaze here. Let me elaborate on what Brian said.

A lot of our customers also restore just for fun or out of convenience. But yea, if you're one of the ones that's doing it out of necessity after a crash or theft, knowing how it works makes everything go a lot smoother.

[roasm]

I back up to a local external drive, and also to BackBlaze. External drives are so cheap these days that the ease of handling your restore yourself is worth it to me. BackBlaze is just an offsite back up so when the house is burning down, I can grab the kids instead of pictures of the kids.

[theandrewbailey]

My alternative: buy 2+ external drives and a fire/water proof box. Keep one drive unplugged and in the box when not backing up. Keep other n-1 drives elsewhere and swap with the local occasionally.

I have one at work, and another at my parent's house, more than 100 miles away. In case of a literal blaze and/or high water, even the local drive should be fine. If all drives are gone from a single catastrophe, I figure I (and everyone else) have more serious things to worry about, like what's for dinner.

[driverdan]

Most fireproof boxes are not actually fireproof. In many fires they will just turn into an oven that bakes whatever is inside.

[bluedino]

>> I have one at work, and another at my parent's house, more than 100 miles away.

How often can you swap them?

[theandrewbailey]

As much as I want. I usually swap the work one about every week or two; parent's house is a few months.

[saryant]

That's what I do. I've got local Time Machine backups (which I have restored from in the past). So far I just haven't had to use my Backblaze backups but I think I'll test that this weekend.

[brianwski]

It should take you less than 20 seconds. Here is the procedure. First sign in here:

https://secure.backblaze.com/user_signin.htm type your username, password, and click "Sign In", then....

Look on the left nave for "View/Restore Files", browse to one file you know you changed recently, and check the checkbox by it (and maybe a few other files) and click "Continue With Restore". Done!

Within a few seconds you can go to "My Restores" (on the left side of the web browser) to download the restore!

[dexterdog]

I just assume that anybody who uses backblaze and is serious about their backups just uses it as a true DR. They keep a local backup which will be used for 99% of restores. BB is just for the case where you lose the backup as well. I honestly don't know how they do it at $5/user/month. They must have a ton of very light users to offset the whales.

[brianwski]

> must have a ton of very light users to offset the whales

Brian from Backblaze here. Our new B2 product is priced at half a penny per GByte per month which accurately reflects how much it costs for us to store your data including a small profit for us.

So the $5/month is profitable for us up to a 1 TByte backup. We have about 25 customers with more than 50 TBytes in a $5/month backup, and yes, we lose money on them (which is FINE - they often recommend us to friends with less data). On the opposite end, we have about 20,000 customers with less than 20 GBytes backed up where we are massively profitable on those particular customers. Interestingly enough, my 84 year old father is in that demographic - no digital music, no digital movies, a few digital photos and a Quicken file. Last year he lost a hard drive, we restored his files from Backblaze. :-)

In between the 50 TByte and 10 GByte customers is a big bell curve with the bulk of our customers basically paying for their own backups.

A different way to think about it is the vast majority of people store files inside their laptops and are happy. The maximum size hard drive you can configure in a laptop today is probably about 1 TByte (the 2 TByte laptop size drives are just starting to appear), so by definition we're profitable on people like this. Technical people think everybody is like them and has a 16 TByte RAID array filled with all the Linux ISOs and movies. :-) But really most people have less than a TByte of personal data.

[ghaff]

That describes me as well. I figure Backblaze is a cheap extra pair of suspenders which is both DR and gives me an additional backup made in an independent way. I have pulled older versions of files off Backblaze a couple times when I've needed to but, now, Time Machine pretty much takes care of that.

I also keep a periodic backup in my office as well as various less systematic backups in various ways. I'd recommend anyone keep at least a couple of backups made with different methods. You never feel more exposed than when you need to use a backup and realize you only have one copy of all your data. Hope nothing else fails or you do something boneheaded with the restore.

[rbritton]

I had a drive fail last fall and used the restore portion for the first time. I actually opted to use it over my local time machine backup and am glad I did. It was just a matter of selecting the files online, waiting for it to create the download, and then doing so. Much more painless than my other options.

[pgrote]

The restore process on their consumer process is simple, but takes some time.

You logon or use their app, choose the folders/files you want to restore and await an email from them letting you know when the folders/files are available for download.

[blacksmith_tb]

Or they will send you a thumb drive (or a bigger external), with your files on it. Obviously the use cases are different, one is "oops, I deleted a folder" and the other is "my machine is a heap of slag".

[krzyk]

The only problem with blackblaze is that they are on osx and Windows only. I'm using Linux so the only cheap backup is from crashplan, kudos to them that they support us.