[renaudg]

In case someone at Backblaze is reading this : I had a less than ideal experience in the past couple of days, trying to report through normal support channels a potentially critical bug breaking the "inherit backup" feature in the Mac client.

Here goes : a few days ago the Backblaze Mac client started prompting me to upgrade urgently to the latest version (6.1.0.370). I went ahead and did that but got "Installation could not complete, please contact support" each time. Support advised me to wipe /Library/Backblaze* , reinstall, then inherit the existing backup.

Well, we never managed to get the inherit process to complete : it failed with "ERR_error_unknown" every time. And I started freaking out.

That's where things get interesting : I became frustrated with the canned responses from support and decided to delve into the client logs myself. I quickly found a potential smoking gun (in bzbmenu.log) : a spelling mismatch in an XML attribute name between the server response (support_inherit="true") and what the client was expecting ("ERROR could not read attribute named supports_inherit"). Notice the extra "s" ? Pretty obvious typo and one that could definitely explain the failure (client can't confirm that the selected backup is eligible for inheritance, bails out)

I was rather happy with my findings and shared them in plenty of details with support (including log excerpts), but instead of a "thank you and here's some free months of service !" response, I got "meh, this log file is irrelevant", and worse: no promise to escalate the issue (until I insisted a second time). They suggested updating to the 6.1.0.372 beta, which turned out to have the exact same bug.

Now, I'm left wondering if should try upgrading to 7.0 and inheriting my 6.1 backup from there. But the point of this message is to share my disappointment with this support experience. I went out of my way as a customer and spent a couple of hours researching and finding something potentially useful to the company, but had no appropriate way to report it. I even tried security contacts, but was (rightfully) told it wasn't a security issue. No word on whether they were going to pass it on to the relevant team.

EDIT: I've just confirmed that the same issue is present in 7.0.0.386. Still can't inherit my existing backup.

[atYevP]

Yev here -> I'm reading! If you're able, please send me the ticket number so that I can review that exchange and escalate it. I'm sorry that support didn't appear grateful for your debugging efforts, but I'll make sure that it gets in front of the developers who can examine the issue.

I would update to v7.0 - it does have a lot of minor fixes in there as well (not sure if inherit is addressed, but worth trying).

[eliog]

I also ran into the same exact problem today. I installed Backblaze this morning and tried to adopt a previous backup and got the same error. I just restarted as a new backup and moved on.

But now reading what renaudg mentioned I took a look at the log files and I see the same error:

20191008161659 - ERROR could not read attribute named supports_inherit

The XML is also shown in the log and the attribute is called "support_inherit".

+1 vote for a few free months for renaudg. :)

[atYevP]

Yev here -> confirmed with the devs that the typo you're seeing doesn't affect the inherit - it's a different process. But we'll fix the typo :D

[renaudg]

Wow thanks Yev ! It's #499226

No hard feelings at all against the support guy, who probably has to deal with dubious customer claims all day. But it was pretty frustrating when he told me the log file I looked at couldn't possibly have anything relevant in it, when I had in fact just showed him plenty of relevant things from it :)

[atYevP]

Just spoke with support you'll be getting an update soon (part of the delay is every time it gets updated by the requester it automatically gets moved to the bottom of the queue due to the way they are ordered - it's a weird quirk).

Support should give you more details but it is indeed the case that some of the logs we have aren't tied to every process - so the transmit logs are paramount for debugging inherit backup state issues, but other log files wouldn't be. That said, we should totally fix the typo in the other log files, I've let the devs know about that!

[renaudg]

Well, not trying to turn this into a support forum any further, but I've just upgraded to 7.0 and right now SMS 2FA seems broken ("The verification code is invalid" every time)

[atYevP]

Yev here -> I'll do support wherever I see my bat-signal :D Do you have an alternate method set? If you reach out to support -> https://help.backblaze.com/hc/en-us/requests/new (they have live chat right now) - they can try an alternate method if our primary provider isn't reaching your device!

[Operyl]

It's completely broken here as well, which was really frustrating. I already wasted 117 days of a yearly subscription on one machine because it ran Catalina Beta (which was my fault, I needed a test machine, and I ate the time lost without complaint), but not being able to inherit that back up really sucked afterwards.

[atYevP]

Yev here -> Have you reached out to support about the issue? They are working with our devs to track all of the issues and could use the additional data points -> https://help.backblaze.com/hc/en-us/requests/new. It's working for most users, so we're trying to whittle down the issue.

[renaudg]

FYI I've just confirmed that the issue is still present in 7.0.0.386 (I've seen the same error messages in the log and the inherit process has failed silently)

[atYevP]

Thanks - you should be getting an update soon!

[renaudg]

Yev, thanks for intervening but I’ve just received an update so incredible and unapologetic that I’m tempted to share verbatim excerpts here, and I think you should take a look.

Support is basically telling me that the inherit process on the server can’t handle my "too large backup state’s indexes", and that I’m gonna have to start afresh and reupload 2TB or so from scratch. Not only that, but in order to even be allowed to do this, I’m gonna have to free up my license first by deleting my existing backup, and accept being left without one during the weeks this will take. You’ve gotta be kidding me ?

And I’m this situation in the first place not even because I have a new computer I want to inherit the backup, but because support themselves suggested that I nuke my local backup state and restore it using the inherit feature, to work around the installer being unable to upgrade me to the latest 6.1 release. That’s how it all started.

No apologies given either, no offer to investigate further to avoid that very inconvenient outcome. I’ve been a customer for 7 years but honestly, if I can’t trust your systems to handle the "too large" 2-3TB backup of my MacBook and a couple of external HDDs and I’m supposed to accept that, that suddenly makes me less confident in everything else you do. I hope I misunderstood something here, but I don’t think I did.

EDIT: Ok, reading the email again I may have misunderstood the part where I need to delete my existing backup first. It looks like I can use the 14 days trial to start a new backup, then transfer the license over before the end of the trial. It’s still pretty bad because there’s no way I‘ll have those 2-3TBs reuploaded within 2 weeks, and of course it’s still a massive inconvenience to do so.

[brianwski]

Disclaimer: I work at Backblaze and I wrote the Inherit Backup State functionality.

> inherit process on the server can’t handle my "too large backup state’s indexes"

Yeah, I'm sorry about that and it is on my plate to fix it. The issue is that Backblaze uses a ZIP library that only handles up to 4 GByte zip files. We zip up your "backup state" (the list of files that were backed up) and download it to the client that is "Inheriting". Your "backup state" has exceeded 4 GBytes, which is unusual, but not unheard of (maybe 2% of our customers right now). And it is "on the rise" as more and more customers have more data, plus backup for longer and longer with Backblaze.

The fix is to either link with a new zip library, or sub-divide your "backup state" into 2 or 3 or 4 zip files. Easy enough, but it doesn't help you this week.

> No apologies given either

I am sorry about this. In our support tech's defense, they were CRUSHED this past week by our attempts to get everybody auto-updated in anticipation of the Macintosh OS X 10.15 Catalina release which came out yesterday. So if they were terse it wasn't out of disrespect, it was out of sheer load they were dealing with.

The issue was that if we didn't get everybody to upgrade, anybody that chose to install Macintosh OS X 10.15 Catalina then it would break Backblaze and popup a completely random error dialog that wasn't helpful and didn't solve the problem. By getting people upgraded before Catalina, there are no issues at all and they won't have to contact support.

> It’s still pretty bad because there’s no way I‘ll have those 2-3TBs reuploaded within 2 weeks

To maximize your chances, make sure you turn off all power savings modes on your computer (like don't even let the monitor go to sleep) and make sure Backblaze is set to use 30 threads, and give it LONG periods of time (overnight 8 hours is ideal). You should be able to backup 1 TByte every 24 hours or so, but it can go slower if you don't have an SSD drive, or if your bandwidth is limited. One idea is to take your computer to a location with faster bandwidth, like a school or your work place and leave it there for two or three days, then carry it back home for the incrementals. By the way, Comcast has announced that it literally intends to offer full 1 Gbit/sec service to every last customer in the United States, so another way to go is upgrade your internet for 1 month, then downgrade it later.

> can’t trust your systems to handle the "too large" 2-3TB backup

We can handle the backups, just not the "Inherit" feature for long running backups with large amounts of data. And it's a fairly straight-forward fix for me to fix that, I just need to get to it. I'm sorry you got bitten by this short-coming, I will get it fixed.

[scarface74]

By the way, Comcast has announced that it literally intends to offer full 1 Gbit/sec service to every last customer in the United States, so another way to go is upgrade your internet for 1 month, then downgrade it later.

Comcast is promising 1GB download speeds. Their maximum upload speed is still an abysmal 35Mbps.

[iofiiiiiiiii]

> but not unheard of (maybe 2% of our customers right now).

That seems like a huge amount of customers, no? The income from 2% of customers is enough to pay for several employees' salaries, I bet.

[renaudg]

Hi Brian,

Thanks for your reply, this is what makes HN awesome !

> Your "backup state" has exceeded 4 GBytes, which is unusual, but not unheard of (maybe 2% of our customers right now)

Not to put any extra pressure on you, but given the raw numbers, 2% of your user base affected by this issue sounds like a pretty big deal to me !

> And it is "on the rise" as more and more customers have more data, plus backup for longer and longer with Backblaze.

That's interesting : given that up until now there was no revision history or deleted files retention beyond 30 days, why would my backup state be larger as a customer of many years than if I signed up now, given the same set of files ? Do you keep metadata for all historical file activity even after the files themselves are gone ? Sounds to me more like a backup log than "state" then !

> By the way, Comcast has announced that it literally intends to offer full 1 Gbit/sec service to every last customer in the United States, so another way to go is upgrade your internet for 1 month, then downgrade it later.

I'm not in the US, but I do have pretty good symmetric 5G. Thanks for the advice, 2 weeks might be enough after all (I'll reinstall to reset the clock, for one thing). It's still a significant nuisance in terms of hogging bandwidth and leaving my laptop running 24/7 for days with fans probably going full speed. I just wonder what Backblaze would say to a customer on metered Internet (still common in many places) or simply with a slower uplink (a large part of Europe is still on ADSL/ADSL2 with 25M down / 1M up)

> And it's a fairly straight-forward fix for me to fix that, I just need to get to it. I'm sorry you got bitten by this short-coming, I will get it fixed.

Again, thanks for responding in person, I couldn't have hoped for much better in terms of a technical explanation.

I think I have 21 days left in the backup I want to inherit before the first external disks in it get deleted. I'm actually tempted now to wait a few more days, in case you manage to get this released quickly (especially if it's a server-side fix) :)

Something that could really be improved in situations like these, is that support should have discretion to help out the customer to make up for an issue that's on your side.

For example, putting backup expiration on hold until the bug is fixed (should be really easy actually now that your new offering is implemented : why did they not offer to move me to 1 year retention temporarily ?), issue an additional license so I didn't have to rely on the trial... anything that helps mitigate a problem that's not my fault. Not to mention, throwing in some free months of service for the inconvenience :)

[atYevP]

Yes, unfortunately the re-upload will take some time, but hopefully you can increase threading and get the data uploaded quickly. Understand it's not ideal, but hopefully there's not too much of a time gap between backups.

[sib]

Hi Yev - sorry to jump in here; I've been a customer for a long time and would like to understand the best way to move some particular directory structure from an internal disc to an external disc (both of which are backed up by Backblaze) without causing that entire directory structure to be fully uploaded again. This has happened more than once and it's not pleasant.

[atYevP]

Yev here -> You're saying that when you moved data over without it changing, the data was re-uploaded? It should deduplicate as long as there's been no changes. If you do see that behavior, please ping support so they can investigate -> https://help.backblaze.com/hc/en-us/requests/new

[sib]

Yes, that's exactly what happened. It required re-uploading more than 300GB of data.

Will reach out to support.

[brianwski]

Disclaimer: I work at Backblaze.

> It required re-uploading more than 300GB of data.

What should occur is that it must READ all of the files to make sure it has transmitted them already, which can take hours sometimes, but only a tiny, tiny amount of data is actually transmitted to the datacenter. The client basically shows endless streams of files flowing through it and saying "Currently Backup Up: puppy.jpg" but it isn't really transmitting the files, just verifying the contents haven't changed.

One way to realize it is doing this is watch a network monitor of some kind. Another is if it is going "impossibly fast", like you only have a 10 Mbit/sec upload pipe and it appears to be uploading at 100 Mbits/sec.

[sib]

Yes, makes sense that it has to read them (and presumably calculate a hash and send that hash to your service) to deduplicate; in this case, it took multiple days to complete the process. This is about how long it took to upload the directory in the first case. I did file a ticket after Yev's reply.

[douglasfshearer]

I had this same experience in regards to support interaction. Starting from scratch is a less than ideal process IMO, so much so I have not done it in the hope that a future update fixes this, and I don't have to jump through all the hoops.

[ryanwaggoner]

I just had this exact same experience / issue the last few days.

First, the whole annoying “Backups are broken! You must manually update” pop up every 30 mins. Any time an update requires some special procedure beyond clicking “Install now”, I view it as likely a dev failure. And providing zero context about WHY makes it worse.

Then, the stupid installer fails repeatedly. Just says “installation failed, contact support”. I do and they tell me to delve into /Library and delete the backblaze files there. Feels kludgy. Are they telling thousands of customers to do this? Or is something fucked with my install? Who knows?

Then, the stupid “inherit backup” thing stalls at 50% repeatedly for hours and uses 99% of my CPU. I finally just killed it and resolved to tackle all this mess later.

Honestly, it’s making me rethink using Backblaze at all. Who knows if my backup will even be there when I need it if this is how they run things? (And yes, I know I need to test my backups. I don’t.) I went looking for alternatives today...

[pieterhg]

Same here, it's happening on an older version of MacOS on my dad's Mac Mini if that's any help.

Weirdest is that updates are normally automatic but now it told my dad to download the new DMG on your website, why?

[atYevP]

Yev here -> it may be that something "ate" the auto-update process, could be A/V, a Firewall, or a break in the connection. He can grab the latest build from here -> files.backblaze.com and support can walk him through the process (live chat from 9am-12PM and 1PM-5PM Pacific). Edit -> support (https://help.backblaze.com/hc/en-us/requests/new)