Hacker News new | ask | show | jobs
by macmac 1037 days ago
According to the company the attacker managed to encrypt both the primary and secondary backup systems.
3 comments

Yes, but as a customer your 3-2-1 strategy should include a backup off that cloud. Not the first time, and won't be the last time a cloud provider has a catastrophic data loss incident. Relying solely on your cloud provider for backups is a risk.
You know that after fire in OVH datacenter, they asked their customers to start their disaster recovery plans - and people asked where is such option in OVH admin menu? Not excusing them, but many customers are completely clueless about backups and data security in general.
the 1 in the 3-2-1 should be somewhere on premise or at least not directly reachable from the internet.

Think: ssh cron job that copies backups from cloud to cold storage

And what if the backup you're copying to cold storage is also encrypted?

How did the saying go? You don't have backups until you've successfully restored from them or something like that. =)

Basically any 3-2-1 system is Schrödinger's backup until you've actually used it.

So you only have 1 backup that you daily overwrite?
You can have X daily backups in rotation and after X days of infiltration they're all garbage because they were overwritten by the malware-encrypted code.

A backup isn't real until you've restored from it. That's why you should restore from backups regularly. Firstly so that you know the process and see it actually works and secondly you can confirm you're actually backing up what you think you are backing up.

We've all set backup scripts and forgot to include new directories or files in the configuration as time went on... =)

No, I think you're misunderstanding.

The parent comment is intending to remind people that many things can happen to a backup after it's done. Backups cannot be "set and forget", as just making the backup isn't enough since so many things can happen after you've taken that backup.

- Bitrot/bitflips silently corrupt your backups and your filesystem doesn't catch it

- The storage your backups are on goes bad suddenly before you can recover

- Your storage provider closes up shop suddenly or the services go down completely, etc

- malicious actors intentionally infiltrate and now your data is held hostage

- Some sysadmin accidentally nukes the storage device holding the backups or some other mistake (to summon the classic, I'm betting there are a few persons who have stories where an admin trying to clean up some leftover .temp files accidentally hit SHIFT while typing

```rm -rf /somedirectory/.temp```

and instead writes:

```rm -rf /somedirectory/>temp```

- (for image level backups) The OS was actually in a bad state/was infected, so even if you do restore the machine, the machine is in an unusable state

- A fault in the backup system results in garbage data being written to the backup "successfully" (If you're a VMware administrator and you got hit by a CBT corruption bug, you know what I'm talking about. If you aren't look just search VMware CBT and imagine that this system screws up and starts returning garbage data instead of the correct and actual changed blocks that the backup application was expecting)

Basically, unless you're regularly testing your backups, there isn't really any assurance that the data that was successfully written at the time of backup is still the same. Most modern backup programs have in-flight CRC checks to ensure that at the time of the backup, the data read from source is the same going into the backup, but this only confirms that the data integrity is stable at the time of the backup.

Many backup suites have "backup health checks" which can ensure the backup file integrity, but again, a successful test only means "at the time you ran the test, it was "okay". Such tests _still_ don't tell you whether or not the data in the backup file is actually usable/not compromised, it only tells you that the backup application confirms the data in the backup right now is the same as when the backup was first created.

So the parent post is correct; until you have tested your backups properly, you can't really be sure if your backups are worth anything.

Combine this with the fact that many companies handle backups very badly (no redundant copies, storing the backups directly with production data, relying only on snapshots, etc), and you end up with situations like in the article where a single ransomware attack takes down entire businesses.

If the data you’re reading is encrypted, you’re still screwed.
A 3-2-1 backup strategy involves keeping three copies of your data, stored on two different types of media, with one copy kept offsite for disaster recovery.

you are still supposed to have multiple backups =)

Incremental backups and alerts on large deltas seem like a good idea

(I mean, on large deltas anywhere)

Backups need an air gap.
I often have people complain after comparing my works instance pricing to other cloud providers...

Then try to explain that rotating a few dozen TB of data offsite to cold offline storage every week isn't cheap. Because unlike some vendors, we take pride in data integrity and ensuring that our DR plan is actually.... you know, recoverable :P

If they are not incremental but append only, an air gap is not strictly needed and can be used as an additional safeguard performed less frequently because of manual overhead. The crux of the matter is to assume the main system has been compromised and preventing overwriting existing data.
I would not agree with this. Append-only file systems and storages aren't a bad idea and definitely help with accidental overwrites, but these systems have been punked quite frequently in many ways, and I've worked with backup companies that home-rolled their own append-only backup implementations.

It didn't stop attackers from using extremely common ways to punk the systems even under the best circumstances for the systems. A forgotten password gets leaked, using the backup applications/storage system's own encryption schemes against the victims, just deleting entire volumes or compromising the OS on the systems, the list goes on.

I wouldn't consider append-only an anti-ransomware technique, it just stops one of many common ways of compromising data. This is good, but I wouldn't rely on it to protect against even a run of the mill ransomware scheme.

... until the next update to these viruses.

To utterly destroy an organisation you don't erase or encrypt their data. You change it. Slowly. A little by a little. A birthday here, a name there, a number ... Using the normal ways to change this data. In this way you can go undiscovered for years, employees get blamed for making stupid errors for a LONG time and there is absolutely no way to fix things, no matter what the backup strategy is.

But for ransomware there needs to be a hope of restoring the data. In this case the value would need to be more oblique.
The ransomware gang buys put options on the victim’s stock. Sabotage-backed options scams have been around for a long time.
Doesn’t matter.