Hacker News new | ask | show | jobs
by davemtl 1799 days ago
Yes, test your backups regularly.

When I worked for a large insurance firm, we would run drills every 6 months to perform off-site disaster recover and operational recovery tests to validate our recovery processes. Everything was tested from WAN links, domain controllers, file backups, mainframe recovery and so much more. We were more or less ready for a nuke to drop.

Obviously this costs money, but if you're an insurance firm, not being able to recover would cost way more than running DR and OR recovery drills every 6-12 months.

1 comments

Why would some companies be so diligent while others get caught with their pants down? Can we tell which is which? Might be a good etf to invest in.
Typically only companies that had a disaster happen to them or their customers (like that insurance) will have the institutional awareness. All the rest will file the risk somewhere with alien abductions and toilet paper shortages. When you tell them what could and will happen they will just shrug it off like you are trying to sell them useless bs.
> When you tell them what could and will happen they will just shrug it off like you are trying to sell them useless bs.

Exactly. It's that mentality which drove me to small-scale contract IT work for smaller "mom and pop" organizations. Give them a fair price and do good work and most of them are happy to have your services, treat you with respect, and are often more than happy to trade knowledge and services for equivalent exchange of same. This can lead to much "win /win/everybody wins!"

And if you take a contract that plays out in an unsatisfactory way, it's easy to simply turn down further contracts from the one problematic customer. More time to give your loyal customers, or hunt down a better customer to replace the bad one. ;)

I think there are 3 stages:

Inexperienced, will buy anything that sounds good and trustworthy, no matter whether snakeoil or real deal, because they don't know better. That is most mom&pop shops.

Burned, will buy/do nothing, because when they were inexperienced they were sold/told crap. Now they trust no-one and also think they can save money.

Experienced, when they had a real disaster in the burned stage, recognized their lack of proper tools and manpower as a reason. Now they try to evaluate suggestions properly through inhouse expertise. Only possible if large enough.

> Inexperienced, will buy anything that sounds good and trustworthy, no matter whether snakeoil or real deal, because they don't know better. That is most mom&pop shops.

Since switching to contract IT work and coming in much more direct contact with "mom & pop" shops than I did in prior years, I've come to realize that most "mom & pop" shops are far more business savvy than they're often given credit for. They mostly just don't have access to any sort of fair and reasonably priced IT folk who ain't tryin' to scam them outta house and home.

I've found that by offering that fair price and quality work, I can gain a level of loyalty that results in me not even needing to advertise my services to have more than enough work and profit to keep me goin' and happy with my career choice. "Word of mouth" is by far the best advertising you could ever ask for anyhow… Nothin' beats trust for generating "brand loyalty" and return business.

> Experienced, when they had a real disaster in the burned stage, recognized their lack of proper tools and manpower as a reason. Now they try to evaluate suggestions properly through inhouse expertise. Only possible if large enough.

I've come across these folk as well. They also tend to be able to recognize instantly when they're not bein' taken advantage of. This type has always been a good loyal customer type worth putting in a bit of extra effort for, too. Having been "burned" before, they recognize the value of payin' a fair price to an honest hardworkin' tech.

> Burned, will buy/do nothing, because when they were inexperienced they were sold/told crap. Now they trust no-one and also think they can save money.

The saddest example of the three, because they'll continue to suffer because their trust had been abused.

Money and time. Throughout my career, there's never been a moment where we're like "All right, let's sit down and assess where we are", or a "Ok we're finished with software engineering, let's do some chaos testing". There's always something that seems more important to do.

I'm now convinced most people are overworked and most SWE projects are overcommitting. I mean I'm currently the sole responsible for two codebases of nearly 300K LOC total, rebuilding the one into the other. At my previous jobs this would involve a fully staffed team of 4+ engineers, tester, product owner, etc - and they could probably use more.

Considering it was almost 20 years ago, just as the Internet was starting to take off and it certainly pre-dates things like Cloudflare, things like this were pretty mandatory. Couldn't tell you if it's still the case, but it did make me appreciate having a good DR and OR plans if the nukes did drop.
> Might be a good etf to invest in.

Nah, it gets way outperformed by the "too big to fail bailout-monkey" ETF.

Unfortunately you need political connections to know the composition of that ETF.

Just go through Cloudflare's list of customers.
You would hope that an insurance company would be good at assessing risk.