[jl6]

I have a feeling there is a very short security-hygiene checklist that, if followed, could prevent the vast majority of the ransomware attacked that we have seen in the last few years.

* Keep all systems up to date with the latest patches.

* Have a DR plan and test it regularly.

* Make frequent backups, verify them, and keep them offline.

Historically organizations have been so bad at backups that the advice has been to automate them as much as possible, to try to ensure that a recent backup at least exists. But I am increasingly of the opinion that the next level of backup maturity is to dial back on the automation and invest manual effort in airgapping the backups.

Fully automated backups are necessarily part of the software attack surface.

If you have to hire more ops people to rotate tapes by hand every day, that will have to be a cost of doing business safely.

[dimastopel]

It looks that the majority of recent ransomware is stealing and publishing private data, instead of blocking access to data. So DLP like solutions are required in addition to posture management.

[ransom1538]

I would like to also add: A system to lower privileges based on last use.

Companies often have IAM/ssh/keys all over the place. If you centralize things to IAM you can lower permissions based on their last use. EG. A frontend dev needs access to GCP to configure things in firebase. This frontend developer hasn't used these IAM permissions in 3 months. This persons IAM permissions should automatically have these permissions removed.

Probably one of the easiest yet most powerful thing to implement in cloud sec ops AND probably never done.

https://cloud.google.com/iam/docs/recommender-managing

Example script to automate it: https://github.com/james-ransom/auto-apply-gcp-iam-recommend...

[roel_v]

It's easy to be healthy, just eat right, exercise more, sleep enough and lower your stress levels. Tadaa! The secret to being healthy!

[Veserv]

You are both correct and incorrect. By following simple procedures you can likely stop the majority of ransomware attacks that have occurred recently, but that is because most of the ransomware attacks were likely done with a budget on the order of $1k-$10k since that is all you need to get a $1M payout from these organizations. No point in running a mission impossible style attack when walking in the front door works just as well.

The problem is that they are getting $1M payouts on a $10k budget. That is a staggering ROI of 100! If you could magically improve the security of every system on the market by 1000% you would wipe out the current forms of attack, but it would still be insanely profitable to run $100k attacks to get $1M payouts. To actually stop attacks from continuing to escalate exponentially at their recent pace of >100% per year that any VC darling would be proud to achieve, you need to make it cost more on average to attack than they can get.

We are literally orders of magnitude away from that in the average case at current returns. And even worse returns per attack keep escalating. Just 4 years ago during WannaCry the ask was $300 per computer which can be a painful chunk of change for an individual which is who most ransomware attacks were targeting before, but nothing for any company. They were attacking companies for ~$10k payout and still making enough money to expand their operations doing it.

As the focus has moved to industry the payouts have increased exponentially since there are many companies whose operations are so valuable that they are willing to pay millions or tens of millions or even hundreds of millions per day. At those payouts there are 0 commercial IT systems that can make attacks unprofitable. So, when those attacks become the ones with the best risk-adjusted ROI you better believe they will occur. And when the attackers have a $10M budget simple defenses and techniques that worked on $10k attacks will not work because the attackers will have literally 100,000% more resources at their disposal in much the same way that defenses that work against a rock thrown at 10 m/s do not work against a ICBM traveling 1000x faster at mach 30.

So yes, simple mitigations would stop the simple successful attacks now, but do not solve the actual problem that it would still be profitable to attack even if they were all implemented everywhere since payouts are so much higher than cost.

[golergka]

To be honest, it seems like a lot of part-time hobby projects created by single engineers have better security practices than whole government agencies.

[jl6]

And there’s a good reason for that - security is a human activity as well as a technology, and the more people involved in an organization, the greater the opportunity for miscommunication and diffusion of responsibility.

It’s easier to get it right when you can put your arms around the whole thing.

[sumtechguy]

Very good points. I would add these as well.

There is another facet to all of this. Money. Just plain old money. It takes time and money to buy and maintain this sort of software.

The 'hobbyist' also has plenty of time and access to the tools. Whereas an org may only have so much budget for it. Which in effect restricts time to do it, and or how many people you can pay to do it. Also depending on the org you may not even have access to the correct tools and documentation.

From a pure user 'end point' usage the security stuff is either in the way or 'just works'. Fixing security is background and does not get you anything new. So it often gets forgotten or downgraded in a budget game for something more shiny as the user lets out their inner verruca salt.

[CalRobert]

Can't speak to the HSE specifically but when I see government jobs come up the pay is vastly lower than private sector.

[alex_anglin]

The Critical Security Controls [1] are a good place to start. Alas, it's neither free or necessarily straightforward to implement them - which is why breaches persist.

[1] https://www.cisecurity.org/controls/cis-controls-list/

[LinuxBender]

I would like to add a couple of ideas to the list.

* Also ensure your Production and DR do not use the same automation, or that there is full segmentation in your automation so that if automation goes sideways, or is compromised, your Production and DR are not simultaneously blown away or encrypted.

* If you can't keep backups offline, at least write them to a write-only destination and/or have an enforced vaulting policy that keeps {n} copies in multiple locations and can't even be deleted by super-users. Deletion must require multiple VP's using MFA to log into a thing and "turn a key" so to speak.

[gregsadetsky]

I typically work in situations where the entire data to be backed up (file storage, database) is on the order of 10-100Gb. The projects I’m working on don’t fit the high profile of a Colonial but I’d rather err on the side of safety.

Is there a service that could regularly fetch data from s3 or even connect to postgres, and regularly send a physical copy of the data by mail?

Does it make sense to offer airgapped backups as a service to smaller companies? Over mail?

[tomjen3]

Why not just buy a tape drive and a few tapes? They are offline and air gapped the moment they are out of the machine and, if you have a small company, they can be stored in the owners house.

That gives you quick retrieval of of-site backups.

The only reason I haven't done something like that for all my personal data is that tape machines are terribly expensive. Tape drives are pretty cheap.

[gregsadetsky]

Yeah, that definitely sounds reasonable.

I was hoping for something SaaS-like that would be automated (so that an external company would be responsible to not forget to do the backups) and no entry cost. As you say, tape drives are expensive.

I was starting to imagine how automable it would be to have machines that downloaded and encrypted data, and small robots popping 128/256Gb (up to 1Tb) SD cards in and out, and even dropping them in envelopes with labels automatically printed out. Then the envelopes would be dropped into a chute as the outgoing mail :)

One obvious issue is that an 128gb card is about $30, so sending one every day would be too expensive. And if you sent one once a week that would mean up to 6 days of lost data.

Then there’s the issue of having access to so much customer data — this imaginary backup company would itself become a potential liability if it was hacked.

Would small companies even be interested?

[tomjen3]

To clarify. a tape reader is about 5k, a tape is cheaper than a hard drive at the same storage capacity and can be stored for a long time without issue.

[galangalalgol]

If you use mail consider if your restore time will be less than your acceptable down time.

[hanselot]

This can be avoided by offering a same-day backup restoration engineer deployed on-site with your last X backups? In fact, this sounds like exactly the kind of thing my managers would want.

[hnick]

> dial back on the automation and invest manual effort in airgapping the backups

Can we please call it The Department of Redundancy Department?

Jokes aside it seems that the DR, backups, and system images (i.e. installation including patches) that you mention are all related and it could make sense to dedicate a role or team to it. We split out things like networking and security into their own teams when we want them to be taken seriously.

[xen2xen1]

Complete, tested tape backups would cure many, many ills. They're out of fashion, but..

[xienze]

The bigger IMO problem with ransomware attacks isn't necessarily that they're holding your data hostage, it's that they can and will publish it. You might be able to tell them to kiss your ass because you have backups, but then they'll publish that information. It's a bit more of a rock-and-a-hard-place situation than most people realize.

[ransom1538]

Eh. From my understanding the people that pay do fine. As sick as it is, these crews following through is good for business. These crews are making tons of cash. If word gets out they don't unencrypt and do publicly publish - people will just stop paying: period. Hell. Some of these crews have a help desk. [1]

https://www.macworld.co.uk/cmsdata/features/3659100/how_to_r...

[steve_gh]

This is a public health service. It's paid for out of taxation.

[headmelted]

Which if you pay the ransom, means also relying on the word of the people that are actively extorting you.

Scary, scary place to be. Especially for a health service.

[xienze]

> Which if you pay the ransom, means also relying on the word of the people that are actively extorting you.

As weird as it sounds, reputation matters for these guys. If you have a track record of taking the money and publishing data anyway, no one is ever gonna bother paying you in the first place. Why would they? Your data is gonna get published no matter what, may as well save the ransom money.

[perl4ever]

You can flip it around (if you're a pessimist):

  1. If you *don't* pay, then you know bad things will happen.

  2. So you might as well pay, regardless of their reputation, because your chances are strictly better even if they are nearly nothing.

  3. Knowing that, there is no incentive for them to maintain a reputation by honoring the ransoms.

This seems like a stable equilibrium.

[grumple]

Tape backups are ok but still mean significant operational downtime because recovery from tape is slow. This is better for long term data storage than rapid recovery.

For recovery, you need more accessible backups. And to test your backup plan.

[rjmunro]

The time to restore from backups after a ransomware attack is more about figuring out how they got in and closing any back doors then cleaning out the existing systems, applying the latest security updates etc., rather than actually restoring the backups from tape or whatever.

The last thing you want is for your backup to restore whatever back door they installed a few weeks before they launched the actual attack, or to leave the unpatched system (or whatever it was) open and immediately have the attackers encrypt all your files again.

[jl6]

High end tape reads sequentially faster than a typical spinning HDD, so it should be possible to design for rapid full restores. Rapid restores of specific files (e.g. to recover from accidental deletion) would be slow from tape though, so are probably best served by online snapshots rather than the DR backup.

[raffraffraff]

I would be amazed if the Irish health service had advanced beyond tape storage. I mean primary storage. (I'm Irish btw)