[chriscjcj]

It's been quite a few years since I did this kind of stuff for a living, so this may be an antiquated notion...

"In my day," desktop computers saved their files to a server. That server would get backed up daily. The backup tapes/drives would be stored offline and rotated to an offsite location. (Back then you were more concerned about the building burning down than a ransomware attack.) The same would be true for any apps running on servers; their data/databases would be backed up daily and the tapes/drives used for backup would be stored elsewhere.

What is this old guy missing? If a process like this were in place, nearly all of their data would be intact. Yes, it will take some time to do a full restore and you will be missing some amount of data that was created since the last backup. But it's survivable in many cases. And you're not negotiating with criminals.

[PeterisP]

The big change is that many places now send data to an offsite location (or cloud!) through a network instead of physically moving tapes, and the attacker can often use the same network connection to destroy backups.

[scarmig]

I think most backup providers (e.g. rsync.net) allow and encourage read-only backups.

The bigger issue is that nowadays organizations have lots of interdependent systems, and if you seize the data of one, you basically cripple the entire organization. So for each system you need to institutionally require both backups and backup testing procedures, which is easier said than done.

[mr_mitm]

The article does not say anything about Oakland negotiating. They may just be in the "it takes some time" phase at the moment. Tapes are not exactly the fastest medium.

Plus, you may want to determine the exact time at which you were compromised, or else you'll be restoring potentially tainted backups. Depending on how well you're organized that alone will take quite some time, especially considering that your logs may be encrypted as well. Sometimes you don't even know how to contact everyone, because your comms are down, too.

Sure, if you do everything right and adhere to all the best practices, it won't be that big of an issue. Just don't forget about the amount of legacy crap and budget constraints many orgs have to deal with. That comes with many pitfalls and a lot of opportunities to make a mistake.

[voidwtf]

This right here, the company I work for did some back of the napkin calculations for restoration times for archival storage (tape) and determined that it’d take us 2 days to restore minimum and up to 7 days to restore if the queues were full. This was with Azure and a relatively small amount of data (around 6TB).

We’re using their immutable storage option, with a 60 day window with multiple rotation intervals, and just biting the bullet on the cost of cold storage vs archival because of how slow tape is.

I could definitely see a larger entity having significantly more data and the restoration process can’t even start until they finish triage. No point in restoring until you know the source of the intrusion or at least have a plan to prevent it from recurring.

[ttt3ts]

For regular backups of 6TB why would you even consider tape? Seriously asking because I have priced tapes and the break even point vs hard drives is high.

[voidwtf]

Because we’re not paying for the tapes, Microsoft is.

Their archival storage tier is extremely cheap, but has a high retrieval cost and a very slow retrieval time. If you wanted to keep 1+ year full and incremental backups of a large amount of data it’d be a no brained. If you’re only keeping 60 day full and incremental than the retrieval cost and time doesn’t pan out cost wise IMO.

[scohesc]

You also have organizations that have certain retention periods - say for example, keep all data for 6 months.

If your ransomware stays resident in your systems for 6 months, any backup you recover from ends up being infected and can potentially be considered useless to restore from unless you're very careful in how and what you restore from.

[alephnerd]

A lot of organizations also don't have the money or processes in place to manage backups. It's a huge cost outlay and in cash strapped SLGs, it simply ain't happening - especially when any half decent talent can make way more money working remotely for companies that respect Engineering.

[908B64B197]

That sounds completely self-inflicted. What are they spending their money on? Not Oakland, but across the bridge, last I heard, 16 millions for a few tents [0].

[0] https://www.nbcbayarea.com/news/local/san-francisco-paying-1...

[alephnerd]

Debt. City of Oakland has has a budget deficit of around $100-200m a year for the past 10 years at least.

[aeternum]

There is no longer any discipline around filesize and data. People regularly attach huge files and have terabytes of useless data in databases. This makes daily backups prohibitively expensive for most orgs.