[comprev]

I had a client which ran a $300M/annum business off a handful of servers with zero redundancy. Their PII/GDPR data was unencrypted on a single spinning disk.

Any advice about risk mitigation fell on deaf ears.

It was the most terrifying 6 months of my career and taught me a harsh lesson in risk assessments of potential clients.

[mattzito]

My first job was at a tiny ISP during the dialup->DSL/cable transition, and I got put in charge of setting up the very first RAID array they had, which was a RAID-5 hardware array (SCSI, natch). It was a 4 drive array, we moved most of our fileserving over to it, next project was to set up a hot standby for the server.

Of course, that was when DSL started to become a lot more common, and business started to suffer. Money got a little tighter, and after a month or two we lost a hard drive in the array - no problem, grabbed the spare, swapped the drive. Then I asked for a replacement spare - "sorry no, can't afford a spare drive, let's wait a few weeks"

Two weeks later, another drive fails - and now we have no spare. I go to the CEO and ask for his amex to overnight a replacement drive, and he says, "Ehhh, no", and I said, "well, you know, if we lose another drive, we'll lose the whole array", and he said, "Yeah, but what are the odds we lose another drive in the next few months?"

I left that job two months later, constant nightmares in the meantime. A month and a half after that I heard from a former colleague that indeed, another drive had failed, and turned into a four day downtime for the entire ISP.